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I.  INTRODUCTION 


A.  BACKGROUND 

1.  Deficient  Contractor  Business  Systems 

In  the  world  of  Department  of  Defense  (DoD)  acquisition  and  contracting, 
managing  costs  is  critical  to  sustaining  program  success  and  preventing  misused 
resources  in  government  contracts.  According  to  the  Commission  on  Wartime 
Contracting  (2009b),  “[contractor  business  systems  and  internal  controls  are  the  first  line 
of  defense  against  waste,  fraud,  and  abuse”  (p.  1)  and  are  essential  to  managing  cost  risk 
to  the  government.  In  2009,  the  Commission  on  Wartime  Contracting  found  multiple 
deficiencies  in  contractor  business  systems  and  internal  controls.  As  a  result,  Section  893 
of  the  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2011  mandated  the 
Secretary  of  Defense  to  implement  a  program  to  improve  the  oversight  of  contractor 
business  systems. 

The  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  Clause 
252.242-7005  identifies  six  contractor  business  systems:  accounting,  estimating, 
purchasing,  earned  value  management  (EVM),  material  management,  and  property 
management.  In  the  event  that  any  business  system  demonstrates  a  significant  deficiency 
in  which  it  is  unable  to  produce  reliable  cost  management  data,  demonstrating  risk  to  the 
government,  contractor  payments  of  up  to  5%  can  be  withheld  for  each  deficient  system 
(DFARS,  2013).  Although  the  DFARS  business  system  rule  identifies  six  systems,  this 
paper  focuses  solely  on  the  Earned  Value  Management  System  (EVMS). 

The  next  section  introduces  the  role  of  EVMS  within  the  Defense  Contract 
Management  Agency  (DCMA)  and  its  definition  of  a  significant  deficiency. 

2.  Earned  Value  Management  System 

EVM  is  the  DoD’s  primary  tool  for  measuring  acquisition  program  cost,  schedule, 
and  performance.  EVM  data  are  measured  against  the  contract’s  performance 
measurement  baseline  to  monitor  progress  and  ensure  the  least  cost  risk  to  the 
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government  (Snider  &  Dillard,  2008).  Mahoney  and  Rego  (2011)  described  EVM  as  “a 
systematic  approach  to  the  integration  and  [measurement]  of  cost,  schedule,  and 
perfonnance  on  projects,  [which]  provides  an  early  warning  system  for  potential  threats 
and  opportunities”  (p.  67).  Correspondingly,  the  EVMS  validation  done  by  DCMA  serves 
as  an  assessment  of  EVM  data  to  certify  that  processes  are  in  accordance  with  the  32 
EVMS  guidelines  as  issued  by  the  American  National  Standards  Institute/Electronic 
Industries  Association  (ANSI/EIA)  748  (DCMA,  2011).  Dibert  and  Velez  (2006)  stated 
that  the  guidelines  provide  a  practical  approach  to  effective  project  management. 

Understanding  EVMS  compliance  and  deficiencies  first  requires  a  working 
knowledge  of  how  the  process  is  monitored,  annotated,  and  corrected.  The  next  section 
discusses  the  DCMA’s  EVMS  role,  EVMS  surveillance  procedures,  and  the 
consequences  of  non-compliance. 

a.  Defense  Contract  Management  Agency  (DCMA)  EVMS 
Surveillance 

The  DCMA  functions  as  the  DoD  Executive  Agent  and  Compliance 
Authority  for  EVMS  compliance  (DCMA,  2012a).  The  DCMA  conducts  EVMS 
surveillance  upon  contract  award  for  all  contracts  valued  over  $20  million,  which  contain 
the  EVMS  Federal  Acquisition  Regulation  (FAR)  or  DFARS  clause  (DCMA,  2012a). 
EVMS  surveillance  ensures  that  contractor  EVM  data  are  in  compliance  with  the  32 
EVMS  guidelines  as  issued  by  ANSI/EIA  748.  However,  the  present  research  project 
focuses  only  on  13  EVMS  guidelines  that  Senior  DCMA  EVM  specialists  have  identified 
as  high-risk  guidelines  that  require  annual  evaluations  as  shown  in  Table  1. 
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Table  1.  DCMA  13  High  Risk  EVMS  Criteria  (from  Dibert  &  Velez,  2006) 


EVMS  Criteria 

13  “High  Risk”  Criteria 

EVMS  Group  #  1:  ( 

Brganization 

#  1 

Define  Authorized  Work  (WBS  Elements) 

#3 

Integrate  the  System 

EVMS  Group  #2:1 

banning,  Scheduling,  and  Budgeting 

#6 

Schedule  the  Work 

#7 

Identify  Products,  Milestones,  and  Indicators 

#8 

Plan  the  Perfonnance  Measurement  Baseline  (PMB) 

#9 

Establish  Budgets  for  Work 

#  10 

Identify  Work  Packages 

#  12 

Identify  and  Control  Level  of  Effort 

EVMS  Group  #  4:  Analysis 

#23 

Analyze  Schedule  and  Cost  Variances 

#26 

Implement  Managerial  Actions 

#27 

Develop  Revised  Estimates  of  Cost  at  Completion 

EVMS  Group  #5:  E 

Revisions 

#28 

Incorporate  Changes  into  Plans,  Budgets,  and  Schedules 

#32 

Document  Changes  to  the  PMB 

b.  Significant  Deficiencies 

The  DCMA’s  annual  surveillance  of  the  13  high  risk  EVMS  guidelines  is 
the  process  that  can  approve  or  disapprove  a  contractor’s  EVMS  (DCMA,  2012a).  Failure 
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to  meet  ANSI/EIA  748’s  standards  for  any  of  the  13  guidelines  results  in  a  significant 
deficiency  and  disapproval  of  the  EVM  system.  DFARS  Clause  252.234-7002(a)  defines 
a  significant  deficiency  as  a  “shortcoming  in  the  system  that  materially  affects  the  ability 
of  officials  of  the  Department  of  Defense  to  rely  upon  information  produced  by  the 
system  that  is  needed  for  management  purposes”  (DFARS,  2013,  Sec  252.242-7005). 

Therefore,  if  a  contractor’s  EVMS  fails  to  meet  one  or  more  high  risk 
guidelines  (1,  3,  6,  7,  8,  9,  10,  12,  23,  26,  27,  28,  and  32),  the  system  is  considered  to 
have  a  significant  deficiency  and  withholding  of  progress  payments  is  required  (DFARS, 
2013).  However,  DCMA’s  Administrative  Contracting  Officer  (ACO)  have  the  discretion 
to  determine  the  level  of  the  Corrective  Action  Request  (CAR)  based  on  the  severity  of 
the  deficiencies  (CAR,  2013).  Table  2  describes  the  different  CAR  levels  and  action 
required. 


Table  2.  CAR  Levels  (CAR,  2013) 


CAR  Level  I 

Deficiencies  are  minor  in  nature  and  can 
be  quickly  corrected 

CAR  Level  II 

Deficiencies  require  time  to  correct  and  a 
corrective  action  plan  by  contractor  is 
required 

As  of  August  16,  2011,  the  new  business  systems  rule  authorizes  DCMA 
ACOs  to  withhold  a  maximum  of  5%  of  contractor  progress  payments  on  all  DCMA 
Level  III  and  above  CARs  until  the  significant  deficiencies  have  been  corrected  (DFARS, 
2013).  Once  the  contractor  makes  progress  towards  implementing  their  corrective  action 
plan,  the  ACOs  have  the  discretion  to  reduce  the  withhold  percentage. 

Lockheed  Martin  is  the  first  contractor  from  which  the  DCMA  has 
withheld  payments  since  implementing  the  new  business  rule  (Fast,  2012).  In  October 
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2012,  the  DCMA  withheld  $46.5  million  from  Lockheed  Martin  for  two  F-35  contracts 
that  were  overrunning  by  approximately  70%  in  cost  and  delayed  in  schedule  (Cappacio, 
2012).  As  of  April  2013,  the  withhold  amount  increased  to  $130  million  as  a  result  of 
failing  to  implement  a  corrective  action  plan  for  the  deficiencies  (Cappacio,  2013).  The 
progress  payments  totaling  $130  million  equate  to  5%  of  funds  spent  by  Lockheed  Martin 
for  both  contracts  (Cappacio,  2013).  The  DCMA  has  no  plans  to  release  the  funds  until 
progress  towards  improving  significant  deficiencies  are  demonstrated  and  in  compliance 
with  the  EVMS  standard  guidelines. 

This  problematic  situation  with  Lockheed  Martin  cautions  DCMA 
officials  to  ensure  EVMS  compliance  and  significant  deficiencies  are  correctly  measured 
in  order  to  implement  the  business  system  payment  withholding  rule.  As  Naval 
Postgraduate  School  Senior  Lecturer  William  Fast,  stated,  “[T]he  financial  impact  and 
materiality  of  the  deficiency  is  difficult  to  quantify-since  it  is  the  inaccurate  and 
unreliable  data  produced  by  the  business  system-that  is  (in  fact)  the  deficiency”  (Fast, 
2012,  p.17).  Therefore,  exploring  objective  and  quantitative  ways  to  identify  a  significant 
deficiency  and  calculating  government  risk  in  dollars  is  the  reason  behind  this  study.  The 
next  section  describes  the  motives  for  our  research. 

B.  PURPOSE  OF  STUDY 

The  purpose  of  this  research  is  to  objectively  and  quantitatively  portray  EVMS 
risk  in  a  way  that  supports  a  monetary  withhold  decision  and  can  withstand  objection  (to 
include  litigation)  from  the  defense  contractor.  In  this  project,  we  hypothesize  that  using 
quantitative  risk  models  such  as  Value  at  Risk  (VaR)  methods  and  simplification  of  that 
business  model  for  use  by  contracting  officers  provides  value  to  DCMA  professionals  in 
the  implementation  of  the  new  business  rule.  The  project  seeks  to  provide  the  DCMA 
with  a  more  defensible  risk  value  model  as  the  basis  for  withholding  contractor  payments. 
The  next  section  describes  the  focus  and  priority  of  the  research  project. 

C.  RESEARCH  OBJECTIVES 

Three  principal  objectives  define  the  project  and  set  the  priorities  of  the  research. 
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1.  Determine  whether  the  13  EVMS  “high  risk”  guidelines  can  be  grouped  with 
respect  to  root  causes  (causality  of  risk). 

•  Evaluate  the  rank  or  natural  order  to  the  potential  severity  of  the 
deficiency  posed  by  these  guidelines. 

•  Assess  the  degree  of  interdependence  or  causality  across  the  13  critical 
EVMS  guidelines. 

2.  Evaluate  which  quantitative  method(s)  can  be  used  to  calculate  risk  value  with 
respect  to  non-compliance  with  both  critical  and  non-critical  guidelines. 

•  Evaluate  which  quantitative  definition  of  “significant  deficiency”  is 
applicable. 

3.  Determine  the  relationship  of  risk  value  calculations  and  findings  of  EVMS  non- 
compliance  with  (a)  probability  of  error,  (b)  magnitude  of  errors,  and  (c)  adverse 
impact  of  errors. 

•  Develop  a  deterministic  rule  set  that  yields  a  consistent  and  repeatable 
finding  of  significant  deficiency. 

The  next  section  suggests  how  relevant  stakeholders  can  benefit  from  the 
research. 

D.  RESEARCH  BENEFITS 

Preventing  and  mitigating  fraud,  waste,  and  abuse  of  taxpayer  dollars  is  the 
ultimate  benefit  of  this  project.  In  2009,  the  Commission  on  Wartime  Contracting  found 
that  the  DCMA  lacked  an  effective  system  that  enforced  EVMS  compliance  among 
contractors  (DCMA,  2009b).  With  the  recent  federal  business  rule  in  effect,  DCMA 
ACOs  now  have  the  withholding  authority  to  motivate  contractors  to  comply;  however, 
the  agency  is  under-resourced  to  implement  the  rule  objectively  (Fast,  2012).  Therefore,  a 
quantitative  risk  model  that  is  simplified  for  use  by  DCMA  contracting  officers  in  the 
implementation  of  the  recent  business  system  rule  can  provide  the  following  advantages: 
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•  Objective  and  quantitative  assessment  of  the  13  critical  EVMS  guidelines 
justifies  appropriate  withhold  amounts  if  necessary,  thus  providing  a 
litigation  safety  net  for  the  DCMA. 

•  The  DCMA  has  a  simplified  risk  analysis  tool,  which  helps  alleviate  the 
issue  of  being  under-resourced  to  motivate  contractors  to  comply  with 
EVMS  surveillance  processes  and  criteria. 

•  A  risk  analysis  model  used  in  assessing  compliance  provides  infonnative 
audit  reports  that  help  ACOs  make  effective  contracting  decisions  well 
beyond  the  decision  to  withhold  payments. 

•  Understanding  root  causes  and  any  correlations  between  the  critical 
criterion  deficiencies  can  provide  decision-makers  with  credible  data  to 
improve  current  EVMS  processes  and  procedures. 

E.  SUMMARY 

The  introduction  included  three  main  topics  of  discussion.  The  first  topic  we 
addressed  was  deficient  contractor  business  systems  and  the  new  business  rule  of 
withholding  5%  of  the  contractor  progress  payment  when  a  significant  deficiency  is 
present.  The  second  topic  focused  on  EVM  business  systems  and  the  DCMA’s  role  in 
monitoring  a  contractor’s  EVMS  for  compliance.  The  final  topic  discussed  the  purpose  of 
our  research  and  the  benefits  we  can  provide  to  the  DCMA.  The  next  chapter  establishes 
a  working  knowledge  by  exploring  EVMS  criteria  in  detail. 
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II.  DCMA’S  13  “HIGH  RISK”  EVMS  CRITERIA 


A.  PREFACE 

This  chapter  provides  a  detailed  description  of  the  selected  13  high  risk  guidelines 
(1,  3,  6,  7,  8,  9,  10,  12,  23,  26,  27,  28,  or  32)  that  the  DCMA  evaluates  for  significant 
deficiencies.  Understanding  the  criteria  in  better  detail  assists  in  determining  root  causes 
of  significant  deficiencies,  which  then  paves  the  way  to  developing  a  methodology  that 
identifies  the  ranking  of  severity  for  each  of  the  high  risk  non-compliant  guidelines. 

Each  essential  criterion  represents  logical  “best  practices”  in  both  the  Department 
of  Defense  and  the  private  sector,  which  would  benefit  any  project  or  program  manager 
(Fleming  &  Koppehnan,  2012).  The  13  guidelines  are  grouped  into  five  categories,  which 
are  (1)  Organization,  (2)  Planning,  Scheduling,  and  Budgeting,  (3)  Analysis,  (4) 
Accounting,  and  (5)  Revisions.  The  Accounting  category  is  excluded  from  the  13  high 
risk  guidelines  because  it  is  a  separate  business  system  under  the  surveillance 
responsibility  of  the  Defense  Contracts  Audit  Agency  (DCAA).  Therefore,  the  13  criteria 
fall  under  only  four  of  the  five  categories. 

B.  EVMS  “HIGH  RISK”  CRITERIA 

1.  Group  1 — Organization 

The  first  category  of  Organization  contains  criteria  1  and  3,  which  clearly  define 
the  range  of  requirements  prior  to  the  project  commencing  (Fleming  &  Koppehnan, 
2012).  Fleming  and  Koppehnan  (2012)  defined  the  following  criteria  as  such: 

a.  EVMS  Criterion  #1:  “ Define  the  authorized  work  elements  for 
the  program.  A  work  breakdown  structure  (WBS),  tailored  for  effective 
internal  management  control,  is  commonly  used  in  this  process  (p. 
3698)” 

The  first  criterion  suggests  the  use  of  a  WBS  to  group  segments  of  work  in 
an  organized  manner  by  hierarchy.  All  deliverables  must  be  documented  and  grouped 
under  a  major  segment  in  the  WBS,  which  ensures  that  project  managers  understand  their 
scope  of  responsibilities  in  order  to  measure  performance  (Fleming  &  Koppehnan,  2012). 
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A  common  issue  that  causes  a  significant  deficiency  in  this  criterion  is  a  poor  definition 
of  the  project’s  range  of  deliverables  (i.e.,  it’s  WBS,  product  specifications),  resulting  in 
wasted  resources  in  regards  to  time,  schedule,  and  performance  (Fleming  &  Koppehnan, 
2012). 


b.  EVMS  Criterion  #3:  “ Provide  for  the  integration  of  the 
company’s  planning,  scheduling,  budgeting,  work  authorization,  and 
cost  accumulation  processes  with  each  other,  and,  as  appropriate,  the 
program  WBS  and  the  program  organizational  structure  (p.  3720- 
3728)” 

The  third  criterion  requires  the  incorporation  of  organizational 
management  in  conjunction  with  the  WBS  to  include  identifying  the  functional  teams 
responsible  for  each  work  package  (Fleming  &  Koppehnan,  2012).  Additionally,  the  third 
criterion  requires  an  information  database  process  to  be  created  for  all  functional  teams 
within  the  project.  Interestingly,  large  defense  contractors  typically  fail  at  satisfying  this 
criterion  due  to  functional  sections  maintaining  opposing  project  goals  and  processes 
(Fleming  &  Koppehnan,  2012). 

2.  Group  2 — Planning,  Scheduling,  and  Budgeting 

The  second  category  of  Planning,  Scheduling,  and  Budgeting  contains  criteria  6, 
7,  8,  9,  10,  and  12,  which  requires  a  management  control  system  that  links  the  formal 
planning,  scheduling,  and  budgeting  of  a  project  into  a  performance  measurement 
baseline  (PMB).  This  group  of  criteria  establishes  a  project  baseline  that  allows  for  a 
formal  means  of  project  discipline  and  assessment.  Fleming  and  Koppehnan  (2012) 
define  the  following  criteria  as  such: 

a.  EVMS  Criterion  #6:  “ Schedule  the  authorized  work  in  a  manner 
that  describes  the  sequence  of  work  and  identifies  the  significant  task 
interdependencies  required  to  meet  the  requirements  of  the  program  (p. 
3781-3788)” 

The  sixth  criterion  requires  a  project  to  establish  a  project  master  schedule 
(PMS)  for  all  projects  within  the  WBS  (Fleming  &  Koppehnan,  2012).  The  PMS  must 
reflect  key  milestone  dates  organized  in  a  manner  to  achieve  optimal  progress  within  the 
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project.  Due  to  multiple  project  changes,  the  most  common  compliance  issue  lies  in 
adhering  to  the  PMS  and  critical  deadlines  (Fleming  &  Koppelman,  2012). 

b.  EVMS  Criterion  #7:  “ Identify  physical  products,  milestones, 
technical  performance  goals,  or  other  indicators  that  will  be  used  to 
measure  progress  (p.  3  796)  ” 

Similar  to  the  sixth  criterion,  the  seventh  criterion  must  identify  tangible 
products  and  deliverables  based  on  the  WBS  in  order  to  measure  earned  value  (Fleming 
&  Koppelman,  2012).  This  criterion  requires  contractors  to  describe  what  products  will 
be  developed  or  delivered  by  a  certain  milestone  date.  However,  the  major  conflict  with 
this  criterion  requires  contractors  to  clearly  identify  the  completed  phase  of  the  physical 
product  or  deliverable  in  accordance  with  the  schedule  (Fleming  &  Koppelman,  2012). 

c.  EVMS  Criterion  #8:  “ Establish  and  maintain  a  time-phased 
budget  baseline  at  the  control  account  level,  against  which  program 
performance  can  be  measured  (p.  3803-3810 )” 

“Initial  budgets  established  for  perfonnance  are  based  on  either  internal 
management  goals  or  the  external  customer-negotiated  target  cost,  including  estimates  for 
authorized  but  unpriced  work.  Budget  for  far  term  efforts  may  be  held  in  higher-level 
accounts  until  an  appropriate  time  for  allocation  at  the  control-account  level.  On 
government  contracts,  if  an  over-target  baseline  is  used  for  performance  measurement 
reporting  purposes;  prior  notification  must  be  provided  to  the  customer.” 

Criterion  8  requires  a  complete  and  fonnal  PMB  in  order  to  measure  all 
components  of  the  project  (Fleming  &  Koppelman,  2012).  Budgets  are  developed  and 
work  is  scheduled  to  be  completed  within  the  scope  of  the  project.  Fleming  and 
Koppehnan  (2012)  emphasized  that  the  PMB  is  critical  to  establishing  a  framework  in 
which  the  earned  value  of  a  project  or  program  can  be  measured  in  accordance  with  time, 
schedule,  and  performance. 


11 


d.  EVMS  Criterion  #9:  “ Establish  budgets  for  authorized  work  with 
identification  of  significant  cost  elements  (labor,  material,  etc.)  as 
needed  for  internal  management  and  for  control  of  subcontractors  (p. 
3841)” 

The  ninth  criterion  is  critical  to  ensuring  the  right  budgets  are  in  place  for 
the  project.  All  costs  to  include  material,  labor,  subcontracts,  and  travel  must  be  included 
in  the  budgeting  phase  (Fleming  &  Koppehnan,  2012).  Furthennore,  Fleming  and 
Koppehnan  stated  that  budgets  must  be  assigned  to  respective  project  sections  and  any 
budget  changes  will  only  pertain  to  authorized  work  as  defined  in  the  WBS. 


e.  EVMS  Criterion  #10:  “ To  the  extent  that  it  is  practical  to  identify 
the  authorized  work  in  discrete  work  packages,  establish  budgets  for  this 
work  in  terms  of  dollars,  hours,  or  other  measurable  units.  Where  the 
entire  control  account  is  not  subdivided  into  work  packages,  identify  the 
far-term  effort  in  larger  planning  packages  for  budget  and  scheduling 
purposes  (p.  3848-3855)  ” 

The  tenth  criterion  focuses  on  clearly  defining  the  work  packages  within 
the  WBS  with  a  clear  distinction  for  the  near  and  long  tenn  (Fleming  &  Koppehnan, 
2012).  Additionally,  established  budgets  containing  numerical  measures  (hours,  dollars, 
etc.)  must  link  to  the  actual  work  function  to  be  perfonned  (Fleming  &  Koppehnan, 
2012). 


f.  EVMS  Criterion  #12:  “ Identify  and  control  level-of- effort  (LOE) 
activity  by  time-phased  budgets  established  for  this  purpose.  Only  that 
effort  that  is  unmeasurable,  or  for  which  measurement  is  impractical, 
may  be  classified  as  LOE  (p.  3870)  ” 

The  remaining  criterion  in  the  planning,  scheduling,  and  budgeting  group 
emphasizes  the  avoidance  of  level  of  effort  (LOE)  activities  such  as  “the  project  manager 
and  staff,  a  field  support  engineer,  guard  services”  (Fleming  &  Koppehnan,  2012,  p.  70). 
These  activities  are  measured  by  the  passage  of  time  rather  than  actual  tangible 
deliverables  and  are  unable  to  be  measured  in  earned  value.  Thus,  Fleming  and 
Koppehnan  stated  that  LOE  activities  must  be  immediately  identified,  have  strict  budget 
controls,  and  be  avoided  to  the  maximum  extent. 
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3.  Group  4 — Analysis 

The  fourth  category  of  Analysis  contains  criteria  23,  26,  and  27,  and  requires 
routine  submission  of  EVM  data  such  as  cost  and  schedule  variances  to  maintain 
effective  project  management  (Dibert  &  Velez,  2006).  Any  variances  from  the  PMB  or 
PMS  must  be  evaluated  and  a  corrective  course  of  action  must  be  in  place  to  minimize 
any  negative  impact  to  the  project  (Dibert  &  Velez,  2006).  Fleming  and  Koppehnan 
(2012)  defined  the  following  criteria  as  such: 

a.  EVMS  Criterion  #23:  “ Identify ,  at  least  monthly,  the  significant 
differences  between  both  planned  and  actual  schedule  performance  and 
planned  and  actual  cost  performance,  and  provide  the  reasons  for  the 
variances  in  the  detail  needed  by  program  management  (p.  4050)” 

The  23rd  criterion  recommends  that  variances  in  schedule  and  cost  be 
reported  on  a  monthly  basis.  Once  a  variance  threshold  is  exceeded,  project  managers 
must  detennine  the  cause  for  a  change  in  the  perfonnance  measurement  baseline  to 
include  developing  corrective  actions  immediately  (Fleming  &  Koppehnan,  2012). 
Furthermore,  this  criterion  applies  to  outside  suppliers  or  subcontractors  of  the  current 
project.  A  main  issue  with  this  criterion  is  a  lack  of  communication  between  the 
contractor  and  subcontractor  in  EVM  reporting  standards  and  managing  variances. 

b.  EVMS  Criterion  #26:  “ Implement  managerial  actions  taken  as 
the  result  of  earned  value  information  (p.  4087)” 

Fleming  and  Koppehnan  (2012)  suggested  a  process  that  ensures 
corrective  courses  of  action  are  implemented  as  a  result  of  exceeding  any  variance 
threshold.  Criterion  26  states  that  projects  may  have  a  set  point  at  which  cost  is  over  or 
under  and  schedule  is  ahead  or  behind,  at  which  management  should  implement 
corrective  actions  to  lessen  any  negative  impact  to  the  project  (Fleming  &  Koppehnan, 
2012). 


c.  EVMS  Criterion  #27:  “ Develop  revised  estimates  of  cost  at 
completion  based  on  performance  to  date,  commitment  values  for 
material,  and  estimates  for  future  conditions.  Compare  this  information 
with  the  PMB  to  identify  variances  at  completion  important  to  company 
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management  and  any  applicable  customer-reporting  requirements, 
including  statements  of  funding  requirements  (p.  4095)” 

The  27th  criterion  focuses  on  the  project’s  estimate  at  completion  (EAC), 
which  should  be  continuously  updated  and  compared  against  actual  work  completed  and 
the  initial  budget  baseline  (Fleming  &  Koppelman,  2012).  Fleming  and  Koppehnan 
stated  that  authorized  work  must  be  budgeted,  and  often  different  project  sections  fail  to 
accurately  estimate  the  final  project  costs,  thus  distorting  the  estimate  of  costs  at  project 
completion.  Most  importantly,  EACs  should  frequently  be  compared  with  the  PMB  in 
order  to  mitigate  cost  issues  as  soon  as  possible  (Dibert  &  Velez,  2006). 

4.  Group  5 — Revisions 

The  last  category,  Revisions,  contains  criteria  28  and  32,  which  require  approved 
changes  to  the  project  in  a  timely  manner  to  allow  for  integration  (Fleming  & 
Koppehnan,  2012).  Fleming  and  Koppehnan  (2012)  defined  the  following  criteria  as 
such: 


a.  EVMS  Criterion  #28:  “ Incorporate  authorized  changes  in  a 
timely  manner,  recording  the  effects  of  such  changes  in  budgets  and 
schedules.  In  the  directed  effort  prior  to  negotiation  of  change,  base 
such  revisions  on  the  amount  estimated  and  budgeted  to  the  program 
organizations  (p.  4134)” 

Fleming  and  Koppehnan  (2012)  stated  that  the  28th  criterion  solely 
focuses  on  integrating  changes  in  a  timely  manner.  Defining  what  is  timely  poses  a  major 
issue  for  this  criterion  due  to  the  type  of  changes  that  would  need  to  be  merged  into  the 
project  baseline  (Fleming  &  Koppelman,  2012).  Additionally,  Fleming  and  Koppehnan 
emphasized  the  importance  of  ensuring  that  all  work  changes  are  immediately 
documented  and  work  value  estimated  into  the  PMB. 

b.  EVMS  Criterion  #32:  “ Document  changes  to  the  PMB  (p.  41 70)” 

The  final  criterion  32  serves  as  an  accounting  measure  to  ensure  all 
changes  are  documented  in  sequence  against  the  approved  PMB  (Fleming  &  Koppelman, 
2012).  Project  changes  are  inevitable;  thus,  a  tracking  system  of  changes  is  imperative. 
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Failure  to  document  changes  to  the  PMB  defeats  the  purpose  of  EVMS  and  EVM  data 
management  (Dibert  &  Velez,  2006). 


C.  SUMMARY 

This  chapter  described  each  of  the  13  high  risk  EVMS  guidelines  in  detail,  which 
offers  a  better  understanding  of  how  to  determine  root  causes  and  severity  of  EVMS 
significant  deficiencies.  The  next  chapter  covers  the  extent  of  quantitative  risk  analysis 
used  in  DoD  acquisition  programs  and  proposed  methodologies  for  this  project. 


15 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


16 


III.  QUANTITATIVE  RISK  ANALYSIS  IN  DOD  ACQUISITION 

PROGRAMS 


A.  PREFACE 

The  implementation  of  quantitative  risk  analysis  in  DoD  acquisition  programs  is 
the  first  step  in  creating  a  framework  and  methodology  of  concepts,  topics,  issues,  and 
data  pertaining  to  our  research  topic.  Key  concepts  that  drive  our  progress  towards 
achieving  our  research  objectives  consist  of  the  following: 

•  Quantitative  Risk  Analysis 

•  Operational  Value  at  Risk 

•  Pairwise  Comparison 

Exploring  these  three  primary  areas  provides  the  link  between  the  research 
questions,  analysis,  and  application  of  the  operational  VaR  model  to  EVMS. 

B.  QUANTITATIVE  RISK  ANALYSIS 

The  use  of  quantitative  cost  risk  analysis  tools  can  be  valuable  in  measuring 
numerical  risk  to  the  government  (Galway,  2004).  However,  quantitative  risk  analysis  is 
rarely  utilized  in  DoD  acquisition  programs  because  the  methods  are  not  easily 
comprehended  by  project  managers  or  integrated  into  programs  (Fast,  2012).  In  fact,  Fast 
discovered  that  quantitative  assessment  is  only  mentioned  once  in  the  Risk  Management 
Guide  for  DoD  Acquisition.  The  current  Guide  (Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (USD  [AT&L],  2006)  focuses  on  the  general  risk 
management  process  to  mitigate  risks,  demonstrating  a  more  qualitative  assessment  of 
risk  (Fast,  2012). 

EVMS  is  used  as  a  risk  management  tool  for  the  overall  program  life  cycle,  but 
the  DoD  Risk  Management  Guide  does  not  address  a  quantitative  assessment  of  the 
EVMS  itself.  Galway  (2004)  practically  linked  project  quantitative  risk  assessment  to 
EVM  by  focusing  on  cost,  schedule,  and  performance  risks. 

Fast  (2012)  recommended  that  the  current  DoD  Risk  Management  Guide  be 
revised  to  include  more  quantitative  risk  analysis  methods.  The  DoD  Risk  Management 
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Guide  (Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD 
[AT&L],  2006)  identifies  three  components  of  risk:  a  future  root  cause,  probability  of 
root  cause  occurring,  and  the  consequence  of  the  root  causes  occurring.  Fast  (2012) 
suggested  that: 

simply  multiplying  the  probability  of  a  risk  event  (expressed  as  a  decimal) 
by  a  monetized  severity  of  loss  would  yield  a  monetary  risk  number. 

Adding  up  all  of  those  monetary  risk  numbers  for  all  elements  of  an 
acquisition  program  would  provide  an  infonnative,  albeit  somewhat 
simplified  quantitative  risk  assessment,  (p.  5) 

Therefore,  identifying  the  cost  to  correct  and  process  a  CAR  level  111  deficiency 
would  lead  to  obtaining  a  monetized  severity  of  loss  in  dollars,  which  will  be  discussed  in 
our  later  chapters.  The  probability  of  a  significant  EVMS  deficiency  occurring  in  a 
DoD  acquisition  project  is  multiplied  by  the  monetized  severity  of  loss  of  that  deficiency. 
This  calculation  can  then  provide  a  quantitative  risk  value  that  could  justify  payment 
withholdings,  which  Chapter  V  discusses  in  further  detail. 

The  following  section  focuses  on  the  Operational  VaR  approach  that  can  be 
applied  in  computing  the  VaR  for  significant  deficiencies  within  a  program’s  EVMS. 

C.  OPERATIONAL  VALUE  AT  RISK 

The  VaR  method  originated  in  the  1960s  when  economist  Harry  Markowitz 
discovered  the  concept  of  measuring  risk  for  hank  portfolio  assets  (Damodaran,  2007).  In 
the  last  few  years,  financial  analysts  and  statisticians  have  brought  forward  various  risk 
measurement  techniques  for  investment  traders  due  to  an  increase  in  trading  activities  and 
uncertainty  in  the  financial  market  (Hendricks,  1996).  In  business,  VaR  is  defined  as 
measuring  “the  worst  expected  loss  that  an  institution  can  suffer  over  a  given  time 
interval  under  normal  market  conditions  at  a  given  confidence  level”  (Butler,  1999,  p.  5). 

The  VaR  method  can  create  value  in  the  acquisition  workforce  by  estimating  the 
government’s  VaR  as  a  result  of  contractor’s  performance.  Thus,  the  VaR  method  is 
proposed  as  the  quantitative  risk  model  for  use  by  government  contracting  officers  when 
justifying  the  withholding  of  contractor  payments  for  a  significant  deficiency  in  the 
contractor’s  EVMS  (Fast,  2012). 
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Jorion  (2007),  a  well-known  author  on  how  financial  institutions  and  the 
insurance  industry  calculate  financial  risk  using  the  VaR  method  describes  Operational 
Value  at  Risk  as  “the  risk  of  loss  resulting  from  inadequate  or  failed  processes,  people, 
and  systems  or  from  external  events”  (p.  495). 

Fast  (2012)  described  operational  risk  as  the  cause  of  corporate  scandals  such  as 
the  Enron  scandal  in  2001  and  the  bankruptcy  of  Lehman  Brothers  in  2008.  Both  cases 
are  an  example  of  poor  corporate  accountability  and  transparency  such  as  a  compliance 
failure,  which  can  be  attributed  to  operational  risk.  Likewise,  operational  risk  in 
government  contracting  can  be  associated  with  the  risk  of  loss  resulting  from  failed 
processes  or  systems  such  as  the  EVMS. 

Operational  risk  is  based  on  the  frequency  and  severity  of  the  expected  and 
unexpected  losses  in  which  two  types  of  loss  distributions  can  occur.  High  frequency/low 
severity  losses  typically  consist  of  small  accounting  errors  due  to  daily  transactions  or 
lack  of  internal  control  (Fast,  2012).  These  are  expected  losses.  Alternatively,  low 
frequency/high  severity  losses  are  considered  disastrous  loss  events  that  pose  the  highest 
risk  to  an  organization  and  could  result  in  bankruptcy  (Fast,  2012).  These  are  unexpected 
losses. 

Jorion  (2007)  measured  the  operational  VaR  using  four  steps: 

1 .  Define  risk  categories  (processes,  people,  systems,  or  external  events) 

2.  Measure  risk  factors  (loss  frequency  defined  as  the  number  of  loss  events  over 
a  set  time  frame) 

3.  Measure  exposure  (loss  severity  defined  as  the  monetary  size  of  the  loss  once 
it  occurs) 

4.  Calculate  risk  through  operational  value  and  expected  loss  (p.497) 

Expected  loss  (EL)  is  how  much  an  organization  can  expect  to  lose  on  average  in 
daily  activities.  In  tenns  of  EVMS,  EL  is  the  amount  lost  as  a  result  of  required 
surveillance  and  follow-up  actions  caused  by  deficient  guidelines.  For  example,  EL  is  the 
government  administrative  costs  of  monitoring  progress  towards  correcting  a  deficient 
contract  and  the  time  lost  to  developing  and  implementing  a  corrective  action  plan. 
Expected  loss  can  be  measured  using  the  following  equation: 
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EL  =  E(n)  x  E(x)(\) 


where 

E(n)  =  Loss  frequency  measured  as  the  number  of  loss  events  over  a  time  frame 
expressed  as  a  decimal 

E(x)  =  Loss  severity  measured  as  the  size  of  the  loss  once  it  occurs  expressed  in 

dollars 

Both  E(n)  and  E(x)  are  two  loss  distributions  that  are  combined  using  the  open 
fonn  solutions  method  in  which  the  mean  of  E (n)  and  E(x)  are  multiplied  to  get  the  mean 
of  the  aggregate  loss  distribution  (Navarette,  2006).  The  mean  of  the  aggregate  loss 
distribution  then  becomes  the  expected  loss. 

EL  =Mean  of  E(n)  x  Mean  of  E(x)  (2) 

Taking  the  kth  percentile  of  the  loss  distribution  or  range  of  expected  losses  then 
becomes  the  operational  VaR  (Navarrete,  2006).  Operational  VaR  can  be  calculated 
using  the  following  equation: 

Operational  VaR  =  k,h  Percentile  of  ELs  (3) 

where 

k,h  Percentile  —  Confidence  level 

ELs  =  Loss  Distribution  /Range  of  Expected  losses  (E(n)  x  E(xf) 

The  unexpected  loss  (UL)  is  the  difference  between  the  operational  VaR  and  the 
expected  loss.  UL  can  bankrupt  an  institution  and  is  the  amount  an  organization  should 
expect  to  insure  for  severe  loss  events  (Navarrete,  2006).  For  EVMS,  UL  is  the  amount 
lost  to  the  government  caused  by  the  potential  severity  a  certain  EVM  deficiency  can 
cause  to  the  contract  performance.  UL  can  be  calculated  using  the  following  equation: 

UL  =  Operational  VaR  -  EL  (4) 

Similarly,  the  expected  and  unexpected  losses  combined  equal  the  operational 

VaR.  For  the  contracting  officer,  EL  can  provide  a  VaR  for  expected  costs  caused  by  a 

significant  deficiency  such  as  administrative  costs  to  monitor  deficiencies.  UL  can 
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provide  the  ACO  with  the  additional  VaR  that  the  significant  deficiency  can  cause  in  the 
most  severe  circumstances  such  as  contract  tennination.  Thus,  the  operational  VaR  can 
be  summarized  as  such: 

Operational  VaR  =  UL  +  EL  (5) 

Overall,  Jorion’s  concept  of  the  operational  VaR  can  be  used  to  detennine  the 
potential  loss  of  a  risky  asset  or  in  terms  of  the  government,  the  potential  loss  in  dollars 
of  a  major  defense  acquisition  program  (MDAP)  regarding  people,  processes,  systems,  or 
events.  A  lack  of  compliance  in  the  EVMS  can  be  attributed  to  the  operational  risk  of  the 
EVM  process;  calculating  the  operational  risk  of  an  MDAP  EVMS  provides  the 
government  with  a  quantitative  tool  to  measure  EVMS  risk.  The  frequency  of  deficient 
EVM  guidelines  can  be  collected  using  historical  DCMA  CAR  data.  Chapters  IV  and  V 
will  demonstrate  that  a  severity  analysis  of  the  high  risk  guidelines  will  result  in  a  rank 
order  of  severity  for  the  13  high  risk  guidelines.  Last,  the  operational  VaR  would  be  able 
to  provide  contracting  officers  with  a  monetary  risk  amount  that  a  deficient  EVMS 
presents. 

D.  PAIRWISE  COMPARISON 

Pairwise  comparison  has  been  utilized  for  many  years  as  a  tool  in  rank  ordering 
solutions  for  a  given  set  of  criteria.  The  basic  principle  of  the  method  is  to  compare  two 
solutions  and  award  each  solution  with  one  point  based  on  meeting  the  established 
criterion;  the  solution  with  the  most  points  wins  (Martin,  2011).  Summers  (2009) 
identified  three  steps  in  utilizing  the  pairwise  comparison  tool: 

1 .  Create  a  table  for  each  criterion  that  has  the  potential  solutions  listed  in  the  first 
column  and  the  first  row. 

2.  Evaluate  each  of  the  solutions  with  respect  to  each  criterion  to  determine  if  the 
row  solution  is  better  (+1  point),  equivalent  (0  point),  or  worse  (-1  point)  than  the  column 
solution. 

3.  Total  the  sum  of  all  the  solutions  for  each  of  the  criteria. 
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Cost 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Honda  Civic 

-1 

+1 

Hyundai  Elantra 

+  1 

+1 

Toyota  Corolla 

0 

-1 

MPG 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Honda  Civic 

0 

+1 

Hyundai  Elantra 

0 

+1 

Toyota  Corolla 

-1 

-1 

Power 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Honda  Civic 

0 

+1 

Hyundai  Elantra 

0 

+1 

Toyota  Corolla 

-1 

-1 

Criteria/Solutions 

Honda  Civic 

Hyundai  Elantra 

Toyota  Corolla 

Cost 

0 

+2 

-1 

MPG 

+  1 

+  1 

-2 

Power 

+  1 

+  1 

-2 

Total  Score 

+2 

+4 

-5 

Figure  1.  Vehicle  Pairwise  Comparison  Example 


Figure  1  demonstrates  that  Hyundai  Elantra  with  +4  points  meets  the  majority  of 
the  set  criteria.  The  drawback  in  using  pairwise  comparison  is  the  inability  to  determine 
whether  a  solution  has  a  higher  weight  over  another  (Summers,  2009).  For  our  project, 
the  pairwise  comparison  tool  provides  a  structured  and  logical  methodology  by 
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comparing  one  guideline  criterion  to  another  in  order  to  determine  the  rank  order  severity 
of  the  13  high  risk  guidelines. 


E.  SUMMARY 

This  chapter  commenced  with  Fast  (2012)  suggesting  that  the  operational  VaR 
can  be  used  to  calculate  risk  in  EVMSs.  ACOs  can  utilize  quantitative  models  in  order  to 
calculate  a  VaR  amount  that  could  justify  withholding  contractor  progress  payments.  The 
operational  VaR  takes  into  consideration  the  frequency  of  a  loss  event  occurring  and  the 
severity  of  the  loss  to  calculate  the  VaR  due  to  failures  from  people,  processes,  or 
systems.  Implementing  the  operational  VaR  as  a  quantitative  DoD  risk  model  can  provide 
contracting  officers  with  a  tool  for  payment  withholds.  Furthennore,  the  pairwise 
comparison  provides  a  method  to  rank  order  the  severity  of  the  high  risk  guidelines  that 
assist  in  calculating  the  severity  of  loss. 

The  next  chapter,  Methods  of  Analysis,  describes  the  approach  to  collecting  and 
analyzing  our  data. 
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IV.  METHODS  OF  ANALYSIS 


A.  ELECTRONIC  CORRECTIVE  ACTION  REQUEST  DATABASE 

Collectable  data  on  EVM  deficiency  reports,  also  known  as  CAR  reports,  serve  as 
a  key  element  for  this  research  project.  The  DCMA  manages  the  electronic  Corrective 
Action  Request  (eCAR)  database  that  consists  of  CAR  reports  ranging  from  levels  I-IV. 
If  a  contractor  is  found  deficient  during  their  EVMS  surveillance,  DCMA  ACOs  must 
issue  a  CAR  to  the  contractor  and  input  the  CAR  report  into  the  database  for 
documentation.  As  a  result,  the  DCMA  has  collected  and  organized  deficiency  reports 
(CAR  reports)  for  over  200  contracts  from  2007-2012.  The  e-CAR  database  organizes 
information  in  the  following  categories:  date  given  to  contractor,  contracts  by 
manufacturing  facility  also  identified  as  Commercial  and  Government  Entity  (CAGE) 
codes,  root  cause  of  deficiencies,  CAR  level,  and  the  deficient  EVM  guideline  per 
contract.  Most  importantly,  the  database  provided  us  with  the  frequency  of  deficient 
guidelines  by  CAGE  code  occurring  within  the  past  five  years. 


CAGE  Code  XX 

Frequency  of  EVMS  Deficient  Guidelines 


GL1  GL3  GL6  GL7  GL8  GL9  GL10  GL12  GL23  GL26  GL27  GL28  GL32 

EVMS  High  Risk  Guidelines 


Figure  2.  Example:  CAGE  Code  XX  Frequency  of  EVMS  Deficient 
Guidelines 
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For  confidential  reasons,  each  CAGE  code  was  recoded  to  keep  company 
information  anonymous.  Figure  2  depicts  an  example  of  deficiencies  that  CAGE  code 
XX  received  within  the  past  live  years.  We  chose  to  determine  loss  frequency  by  CAGE 
code  because  EVM  systems  are  certified  and  validated  by  DCMA  on  a  facility  basis. 

The  frequency  of  each  guideline  as  shown  in  Figure  2  was  divided  by  a  total  of 
1 1  deficiencies  that  spanned  over  five  years  for  Facility  XX.  Table  3  below  lists  the 
calculated  loss  frequency  percentage  of  each  guideline  expressed  as  a  decimal. 
Depending  on  the  total  amount  of  deficiencies  received  at  the  facility,  ACOs  will  obtain 
different  loss  frequency  percentages  for  each  CAGE  code. 


Table  3.  Example:  CAGE  Code  XX  Loss  Frequency  % 


EVMS 

Guideline 

CAGE  Code 
XX 

Frequency 

Loss 

Frequency 

% 

1 

0 

0.000 

3 

1 

0.091 

6 

1 

0.091 

7 

0 

0.000 

8 

1 

0.091 

9 

0 

0.000 

10 

2 

0.182 

12 

1 

0.091 

23 

1 

0.091 

26 

1 

0.091 

27 

1 

0.091 

28 

2 

0.182 

32 

0 

0.000 

In  the  event  Facility  XX  receives  a  first  time  deficiency  in  guideline  1,  all  loss  frequency 
percentages  as  shown  in  Table  3  would  change  correspondingly  based  on  a  new  total  of 
12  deficiencies. 


26 


B. 


METHOD  FOR  DETERMINING  EVM  GUIDELINE  RANK  ORDER 


We  evaluated  the  rank  order  of  severity  posed  by  the  13  high  risk  guidelines  in 
three  sequential  stages.  In  the  first  stage,  we  rank  ordered  the  four  EVM  groups  and  their 
respective  guidelines  from  most  severe  to  least  severe,  based  on  our  analysis  and 
supporting  research. 


Table  4.  EVM  Groups  and  Associated  High  Risk  Guidelines 


EVM  Group 

EVM  Guidelines 

Organization 

1,3 

Planning,  Scheduling, 

6,  7,  8,9,  10,  12 

&  Budgeting 

Analysis 

23,  26,  27 

Revisions 

28,32 

Second,  we  conducted  a  pairwise  comparison  between  the  guidelines  within  each 
group  resulting  in  a  score  of  1  if  the  given  guideline  was  more  severe  than  the  other  and  a 
score  of  0  for  the  less  severe  guideline.  Table  4  outlines  20  pairwise  guideline 
comparisons,  which  are  segmented  by  their  respective  EVM  group. 
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Table  5.  20  Pairwise  Comparison  Combinations  of  Guidelines 


In  the  third  stage,  we  obtained  a  total  score  for  each  guideline  from  the  pairwise 
comparison  method  using  Figure  2  as  our  score  card.  A  point  of  1  or  0  goes  into  each 
blank  cell.  Upon  conclusion  of  the  comparisons,  points  are  totaled  at  the  bottom  row  for 
the  overall  pairwise  score. 
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Using  Table  4  as  a  guide,  we  rank  ordered  each  guideline  within  their  respective 
groups  based  on  their  pairwise  score.  Next,  we  combined  the  ranking  of  the  EVM  groups 
and  guidelines  within  each  group  to  configure  the  overall  ranking  from  1  through  13, 
with  1  ranked  as  the  most  severe  EVM  guideline.  For  example,  if  the  EVM  group 
Organization  is  ranked  as  the  most  severe  group  and  among  its  respective  guidelines  the 
pairwise  comparison  demonstrates  that  guideline  3  is  more  severe  than  guideline  1,  the 
overall  ranking  results  in  guideline  3  as  the  most  severe  EVM  guideline  (rank  1  of  13). 
Similarly,  if  the  EVM  group  Revisions  is  ranked  as  the  least  severe  group  of  the  four  and 
among  its  respective  guidelines  the  pairwise  comparison  demonstrates  that  guideline  28 
is  less  severe  than  guideline  32,  the  overall  ranking  results  in  guideline  28  as  the  least 
severe  EVM  guideline  (rank  13  of  13). 

C.  METHOD  FOR  DETERMINING  THE  GUIDELINE  RISK  FACTOR 

Upon  determining  the  overall  ranking  for  the  EVM  guidelines,  the  risk  factor  for 
each  guideline  was  assigned  using  the  rank  sum  method.  For  research  purposes  and 
simplicity  of  assigning  risk  factors,  we  chose  to  use  the  rank  sum  method  especially  since 
we  will  be  developing  a  rank  order  of  severity  for  each  of  the  13  high  risk  guidelines. 

However,  DCMA  officials  or  the  ACO  can  utilize  other  methods  to  best  fit  their 
specific  manufacturing  facility  or  contract  situation  in  determining  the  risk  factor.  For 
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example,  the  risk  factor  can  be  determined  using  the  direct  assessment  method  in  which 
the  DCMA  would  subjectively  assign  the  weighted  risk  for  each  deficient  guideline 
depending  on  their  assessment  of  guideline  severity  to  their  respective  manufacturing 
facility.  For  example,  the  1st  six  guidelines  could  be  given  an  equal  weight  of  0.05  and 
the  remaining  seven  guidelines  could  be  given  an  equal  weight  of  0.10. 

For  this  research  project,  the  rank  sum  technique  takes  the  sum  of  the  ranks;  in 
this  case  a  ranking  of  13  items  (1+2  +  3+  4  +  5  +  6  +  7  +  8  +  9+10+11  +  12+  13) 
provides  a  total  sum  of  91.  To  obtain  the  risk  factor,  each  rank  item  is  then  divided  by  91. 
The  highest  factor  is  assigned  to  rank  1  and  each  consecutive  rank  item  is  assigned  to  the 
next  highest  factor  and  so  on.  Table  6  outlines  the  rank  sum  method  and  corresponding 
risk  factor.  The  risk  factor  can  be  a  useful  guide  for  ACOs  in  conducting  and  prioritizing 
surveillance  activities.  For  example,  higher  risk  factor  items  could  warrant  more  frequent 
surveillance. 


Table  6.  Ranking  and  Risk  Factors 


Rank  Item 

Rank  Sum 
Calculation 

Risk 

Factor 

1 

13/91 

0.143 

2 

12/91 

0.132 

3 

11/91 

0.121 

4 

10/91 

0.110 

5 

9/91 

0.099 

6 

8/91 

0.088 

7 

7/91 

0.077 

8 

6/91 

0.066 

9 

5/91 

0.055 

10 

4/91 

0.044 

11 

3/91 

0.033 

12 

2/91 

0.022 

13 

1/91 

0.011 

Rank  Item  Sum:  91  Total: 
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D.  SUMMARY 

The  eCAR  database  consisting  of  EVMS  data  for  over  200  contracts  covering  the 
past  five  years  is  an  essential  source  of  data  for  our  analysis  and  computation  of  the 
operational  VaR.  The  eCAR  database  is  critical  to  obtaining  the  loss  frequency  by  CAGE 
code  of  the  high  risk  guidelines,  which  is  required  for  the  application  of  the  VaR  model. 
Establishing  the  rank  order  of  severity  for  the  EVM  guidelines  was  conducted  in  three 
stages:  rank  ordering  the  EVM  group,  utilizing  pairwise  comparison  to  rank  order  the 
guidelines  within  each  group,  and  then  combining  both  EVM  group  and  guideline 
rankings  to  obtain  the  overall  ranking  from  1  to  13.  The  risk  factors  for  each  rank  item 
were  established  using  the  rank  sum  methodology.  However,  DCMA  officials  have  the 
discretion  to  use  other  methodologies  that  best  meet  their  specific  facility  to  obtain  the 
risk  factors,  which  can  be  used  by  ACO’s  to  prioritize  surveillance  activities. 

The  next  chapter,  Analysis  and  Results,  reveals  the  rank  order  of  severity  for  each 
high  risk  guideline  and  the  application  of  the  operational  VaR  to  EVMS. 
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V.  ANALYSIS  AND  RESULTS 


A.  SEVERITY  ANALYSIS  OF  EVMS  GROUPS  &  GUIDELINES 

In  this  section,  we  discuss  our  findings  on  the  rank  order  of  severity  analysis  for 
the  13  EVMS  high  risk  guidelines.  The  rank  order  of  severity  can  assist  ACO’s  in 
prioritizing  surveillance  activities  by  focusing  limited  resources  on  essential  high  risk 
guidelines.  First,  we  discuss  our  analysis  in  determining  the  EVMS  group  rank  order  of 
severity.  Second,  we  describe  the  reason  for  our  selection  in  each  of  the  20  pairwise 
comparison  combinations  of  the  high  risk  EVMS  guidelines. 

1.  EVMS  Groups 

As  described  in  the  introduction,  EVM  measures  a  program’s  cost,  schedule,  and 
performance  by  identifying  measures  to  ensure  a  program  is  in  accordance  with 
ANSI/EIA  748  standards  and  on  track  for  success.  Each  of  the  13  guidelines  belong  to 
one  of  the  four  EVMS  groups  covered  in  this  project.  Table  7  outlines  the  cluster  of 
guidelines  by  group,  which  were  ranked  from  most  severe  rank  1  to  least  severe  rank  4. 


Table  7.  EVMS  Group  Ranking 


Rank 

Order 

EVMS  Group 

EVMS 

Guidelines 

1 

Planning, 
Scheduling,  & 
Budgeting 

6,  7,  8,  9,  10,  12 

2 

Analysis 

23,  26,  27 

3 

Organization 

1,  3 

4 

Revisions 

28,  32 
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The  following  demonstrates  our  reasoning  for  each  EVMS  group  ranking. 

•  Rank  1 :  Planning,  Scheduling,  &  Budgeting 

We  ranked  this  group  as  most  severe  because  establishing  a  PMB  is  the  core 
function  of  this  group,  which  is  also  what  EVM  data  are  measured  against  to  monitor 
program  progress  and  risk  mitigation.  This  group  function  schedules  authorized  work  and 
applies  the  right  amount  of  resources  to  appropriately  budget  the  program;  these 
measures  feed  into  establishing  the  PMB  (Government  Accountability  Office  [GAO], 
2009).  Common  root  causes  of  poor  program  perfonnance  include  issues  with  the 
schedule  guidelines  within  this  group,  such  as  too  many  activities  scheduled  with 
unrealistic  timeframes.  Additionally,  the  eCAR  database  demonstrates  that  the  sum 
amount  of  frequencies  from  this  group  equate  to  48.2%  of  the  total  frequencies  caused  by 
the  13  high  risk  guidelines.  Failure  to  properly  plan,  schedule,  authorize  work,  determine 
a  budget,  and  develop  a  PMB  results  in  significant  deficiencies  by  which  the  program 
will  have  no  baseline  to  monitor  progress.  The  functions  of  this  group  must  be 
established  first  before  the  program  can  proceed  further. 

•  Rank  2:  Analysis 

Analysis  was  ranked  as  the  second  most  severe  group  because  it  requires 
identification  and  analysis  of  cost  and  schedule  variances.  Analyzing  any  variance  within 
the  program  is  critical  to  developing  a  corrective  plan  of  action  to  mitigate  any  severe 
consequences.  This  functional  group  requires  analysis  of  the  reporting  measures  for  EVM 
such  as  cost  variances  (CV),  schedule  variance  (SV),  or  estimate  at  completion  (EAC), 
which  are  critical  indicators  of  program  perfonnance.  Furthennore,  data  accuracy  is  a 
common  indicator  of  poor  program  perfonnance.  Issues  such  as  lack  of  planning  for 
corrective  actions,  multiple  data  input  errors,  and  various  reasons  for  variances  play  a 
role  in  ensuring  that  the  guidelines  within  this  group  are  implemented  to  standard. 
Additionally,  the  eCAR  database  demonstrates  that  the  sum  amount  of  frequencies  from 
this  group  equates  to  26.6%  of  the  total  frequencies  caused  by  the  13  high  risk  guidelines. 
This  group  requires  routine  evaluation  of  the  PMB  and  PMS  as  a  safeguard  against  not 
meeting  critical  deliverables  and  milestones.  Upon  establishing  a  PMB  and  PMS,  routine 


34 


analytical  evaluation  is  essential  to  monitoring  EVMS  data  for  accuracy  and  program 
success. 

•  Rank  3:  Organization 

The  third  most  severe  group  is  organization,  which  defines  the  scope  of  the  WBS 
and  organizational  responsibilities  for  each  WBS  package.  Upon  authorizing  the  WBS  to 
include  ensuring  accurate  data  analysis  of  the  authorized  work,  the  organization  group 
defines  the  range  of  requirements  for  the  program  and  integrates  the  cost  and  scheduling 
data  into  an  organizational  structure.  Additionally,  the  eCAR  database  demonstrates  that 
the  sum  amount  of  frequencies  from  this  group  equates  to  13.9%  of  the  total  frequencies 
caused  by  the  13  high  risk  guidelines.  Organization  of  authorized  work  that  is  accurately 
analyzed  is  essential  prior  to  commencing  a  project. 

•  Rank  4:  Revisions 

We  ranked  the  revision  group  as  least  severe  because  the  requirements  of  this 
group  are  implemented  after  a  PMB  is  established,  data  accuracy  is  evaluated,  and 
organizations  of  WBS  packages  are  conducted.  Revisions  are  a  mechanism  to  document 
and  manage  changes  in  a  timely  manner.  Additionally,  the  eCAR  database  demonstrates 
that  the  sum  amount  of  frequencies  from  this  group  equates  to  11.1%  of  the  total 
frequencies  caused  by  the  13  high  risk  guidelines.  This  group  is  ranked  last  because  the 
sequential  order  of  the  guidelines  within  this  group  can  only  be  conducted  after  the 
requirements  in  the  other  three  groups  are  executed. 

2.  EVMS  Guidelines 

The  pairwise  comparison  combinations  are  only  between  guidelines  (GLs)  within 
the  same  group.  Thus,  our  rank  order  analysis  is  limited  to  only  ranking  guidelines  based 
on  the  rank  of  their  respective  EVM  group.  For  example,  GL  1  could  be  ranked  within 
the  top  five  GLs  in  terms  of  most  severe;  however,  because  GL  1  belongs  to  the 
organization  group,  which  is  ranked  3,  its  overall  rank  might  be  10  of  13. 

Each  pairwise  combination  was  compared  based  on  selecting  the  guideline  that 
posed  a  higher  risk  defined  as  a  measure  of  future  uncertainties  in  achieving  program 
perfonnance  goals  within  defined  cost  and  schedule  constraints.  Each  guideline 
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(significant  deficiency)  was  evaluated  in  terms  of  severity,  which  is  the  consequence  of 
the  future  occurrence  of  that  significant  deficiency.  The  following  sections  discuss  the 
reasoning  for  each  selected  EVM  GL  that  obtained  a  total  score  of  1  or  higher  as  shown 
in  Figure  4: 
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Figure  4.  Pairwise  Comparison  Scores 


a.  EVM  Guideline  1:  Define  Authorized  Work 

In  this  comparison,  GL  1  is  compared  with  GL  3  as  shown  in  Table  8.  We 
selected  GL  1  as  more  severe  based  on  defining  the  range  of  requirements  for  work  using 
the  WBS.  For  tables  8  through  16,  the  highlighted  GL  is  more  severe  when  comparing 
between  the  two  GLs. 


Table  8.  GL  1  vs.  GL  3 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

1 

X 

GL  1  Define  Authorized  Work  (WBS  Elements) 

GL  3  Integrate  the  System 
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The  WBS  is  an  essential  component  of  EVM  and  the  basis  to  correlate 
estimated  costs  and  schedule  with  actual  costs  (GAO,  2009).  A  program  must  have  an 
organized  WBS  structure  before  integration  of  the  system  can  take  place  as  stated  in  GL 
3.  EVM  criterion  3  integrates  all  system  inputs  such  as  costs,  schedules,  deliverables,  and 
the  WBS  structure  to  include  identifying  team  roles  and  responsibilities.  For  example,  the 
F-22  Spiral  2  case  study  showed  that  the  integrated  master  schedule  (IMS)  and  cost 
estimates  were  tied  to  the  Spiral  2  WBS,  for  without  the  WBS,  the  F-22  team  would  not 
be  able  to  integrate  the  critical  inputs  (Dibert  &  Velez,  2006).  Additionally,  the  WBS  is 
essential  to  developing  the  PMB  which  EVM  data  for  the  program  are  measured  against. 
Since  the  PMB  serves  as  the  baseline  for  measuring  EVM,  deficiencies  in  this  GL  are 
linked  with  deficiencies  in  GLs  1  and  6.  In  order  to  develop  a  well-defined  PMB,  there 
should  not  be  a  deficiency  in  GL  1  (development  of  WBS)  or  6  (scheduling  authorized 
work)  because  both  are  required  to  develop  the  PMB. 

b.  EVM  Guideline  6:  Schedule  the  Work 

Table  9  shows  the  pairwise  comparisons  in  which  GL  6  was  selected  as 
more  severe  than  GLs  7,  9,  10,  and  12. 


Table  9.  GL  6  vs.  GLs  7,  9,  10,  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

2 

X 

GL  6  Schedule  the  Work 

GL  7  Identify  Products,  Milestones,  and  Indicators 

3 

X 

GL  6  Schedule  the  Work 

GL  9  Establish  Budgets  for  Work 

4 

X 

GL  6  Schedule  the  Work 

GL  10  Identify  Work  Packages 

5 

X 

GL  6  Schedule  the  Work 

GL  12  Identify  and  Control  Level  of  Effort 
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EVM  criterion  6  was  selected  as  more  severe  based  on  developing  a  PMS, 
which  serves  as  the  source  for  scheduling  authorized  work.  Failure  to  accurately  establish 
critical  milestone  dates  in  an  organized  manner  highly  impacts  the  scheduling  variance. 

In  the  F-22  case,  the  PM  established  a  detailed  PMS;  however,  constantly  changing 
requirements  made  adhering  to  the  schedule  difficult  (Dibert  &  Velez,  2006).  The 
numerous  changes  created  a  negative  domino  effect  for  the  integrated  product  team  (IPT) 
responsible  for  developing  the  work  package  activity  schedules  (Dibert  &  Velez,  2006). 
Scheduling  the  work  feeds  into  the  schedule  variance  (SV),  which  is  a  direct  EVM 
measure  of  success  and  a  deciding  factor  towards  maintaining  or  canceling  a  program. 

The  authorized  work  must  be  scheduled  before  the  budget  or  work  packages  can  be 
established  to  include  FOE  activities  as  described  in  GEs  9,  10,  and  12. 

c.  EVM  Guideline  7:  Identify  Products,  Milestones,  and  Indicators 

Table  7  shows  that  the  following  pairwise  comparisons  identified  GL  7  as 
more  severe  than  GLs  9,  10,  and  12.  GL  7  describes  the  tangible  products  to  be 
developed  in  accordance  with  the  established  schedule. 


Table  10.  GL  7  vs.  GLs  9,  10,  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

6 

X 

GL  7  Identify  Products,  Milestones,  and  Indicators 

GL  9  Establish  Budgets  for  Work 

7 

X 

GL  7  Identify  Products,  Milestones,  and  Indicators 

GL  10  Identify  Work  Packages 

8 

X 

GL  7  Identify  Products,  Milestones,  and  Indicators 

GL  12  Identify  and  Control  Level  of  Effort 

We  found  GL  7  to  be  more  severe  than  its  opponents  due  to  the  difficulty 
in  measuring  the  products  in  terms  of  value.  Furthermore,  contractors  must  clearly 
identify  the  completed  phase  of  the  physical  product  in  accordance  with  the  schedule 
(Fleming  &  Koppelman,  2012).  GL  7  is  a  challenging  criterion  to  meet  especially  in  the 
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field  of  software  development  because  both  the  government  and  contractor  must  agree  on 
the  value  that  each  product  possesses,  which  can  be  complex  to  measure.  In  the  F-22  case 
study,  Lockheed  Martin  experienced  difficulty  in  developing  new  software,  which  led  to 
problems  in  identifying  the  measure  of  success  for  this  portion  of  the  program. 
Furthermore,  F-22  issues  within  this  criterion  resulted  in  special  emphasis  from  the 
government.  The  requirements  as  established  in  GL  7  are  linked  to  GLs  6  and  12  in  that 
identifying  the  products  must  be  in  accordance  with  scheduling  milestones  and  LOE 
activities  must  be  identified  during  the  activities  of  this  phase  as  well. 

d.  EVM  Guideline  8:  Plan  the  Performance  Measurement  Baseline 

In  the  following  comparisons,  GL  8  obtained  a  total  score  of  5  for  being 
selected  as  more  severe  than  GLs  6,  7,  9,  10,  and  12  as  shown  in  Table  11. 


Table  11.  GL  8  vs.  GLs  6,  7,  9,  10,  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

9 

GL  6  Schedule  the  Work 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

10 

GL  7  Identify  Products,  Milestones,  and  Indicators 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

11 

X 

GL  8  Plan  the  Perfonnance  Measurement  Baseline 

GL  9  Establish  Budgets  for  Work 

12 

X 

GL  8  Plan  the  Perfonnance  Measurement  Baseline 

GL  10  Identify  Work  Packages 

13 

X 

GL  8  Plan  the  Performance  Measurement  Baseline 

GL  12  Identify  and  Control  Level  of  Effort 

EVM  GL  8  serves  as  the  most  severe  criterion  based  on  the  critical 

function  of  the  PMB.  The  PMB  establishes  a  framework  to  measure  and  monitor  EVM  of 

cost,  schedule,  and  performance  of  a  program  (Fleming  &  Koppelman,  2012).  The  PMB 

develops  the  budget  and  funds  the  scheduled  work  authorized  for  the  program.  PMB 

integrates  schedule  and  cost  into  one  baseline.  Measurements  outside  of  the  baseline 

signal  focus  areas  for  the  program  management  team.  A  common  indicator  of  poor 
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performance  is  in  developing  PMBs  in  which  budgets  are  unequally  distributed  to  earlier 
scheduled  tasks,  thus  concealing  issues  until  it  is  too  late  to  fix  them  (GAO,  2009). 
DCMA  EVM  specialists  have  stated  that  contractors  tend  to  only  create  a  PMB 
constructed  on  the  base  year  rather  than  the  life  cycle  of  the  program  due  to  the  current 
budget  appropriations.  This  major  issue  likely  leads  to  cost  overruns  and  can  severely 
affect  the  credibility  of  program  officials  and  put  the  program  at  high  risk  for  cancellation 
(GAO,  2009).  Furthermore,  without  a  well-developed  PMB,  there  is  no  foundation  or 
reference  point  to  measure  EVM  data  against  performance. 

e.  EVM  Guideline  9:  Establish  Budgets  for  Work 

In  this  comparison,  GL  9  is  selected  as  more  severe  than  GL  12  because  it 
involves  establishing  budgets  for  the  authorized  work. 


Table  12.  GL  9  vs.  GL  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

14 

X 

GL  9  Establish  Budgets  for  Work 

GL  12  Identify  and  Control  Level  of  Effort 

This  guideline  requires  an  appropriate  amount  of  funding  as  negotiated  for 
the  program;  however,  program  teams  tend  to  obligate  any  excess  funds  for  another 
element  of  the  program.  The  severity  involved  in  inappropriately  allocating  funds 
elsewhere  leads  to  a  deficiency  and  may  likely  affect  funding  in  later  phases  such  as 
integration  and  testing  (GAO,  2009).  Excess  funding  should  be  placed  in  management 
reserve  (MR)  to  mitigate  future  budgeting  risks.  Establishing  budgets  appropriately  from 
the  beginning  is  crucial  to  how  the  program  is  able  to  mitigate  financial  risks  throughout 
the  life  cycle  of  the  program. 

f  EVM  Guideline  10:  Identify  Work  Packages 

Table  13  demonstrates  that  GL  10  was  selected  as  more  severe  in 
comparison  with  GLs  9  and  12  because  it  involves  identifying  work  packages  based  on 
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the  WBS  in  terms  of  budgets,  hours,  and  other  EVM  numerical  measures  for  the  near  and 
long  term  (Fleming  &  Koppelman,  2012). 


Table  13.  GL  10  vs.  GLs  9  and  12 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

15 

GL  9  Establish  Budgets  for  Work 

X 

GL  10  Identify  Work  Packages 

16 

X 

GL  10  Identify  Work  Packages 

GL  12  Identify  and  Control  Level  of  Effort 

Before  one  can  establish  a  budget,  the  work  package  measures  must  be 
identified  to  include  establishing  an  MR  for  future  uncertainties.  In  the  F-22  case,  work 
packages  especially  outside  of  the  current  year  were  difficult  to  identify  due  to  the 
constantly  changing  requirements.  Likewise,  because  criterion  12  involves  identifying 
LOE  activities  that  measure  time  rather  than  tangible  products  that  can  be  measured  using 
EVM,  it  was  rated  as  the  least  severe  in  this  EVM  group.  LOE  activities  provide  no  value 
to  EVM  because  they  do  not  contribute  to  measuring  SV  or  CV;  however,  LOE  activities 
must  be  identified  to  ensure  PMs  do  not  account  for  this  as  EVM  measurements. 

g.  EVM  Guideline  23:  Analyze  Schedule  and  Cost  Variances 

As  shown  in  Table  14,  GL  23  was  selected  as  more  severe  in  comparison 
with  GLs  26  and  27. 


Table  14.  GL  23  vs.  GLs  26  and  27 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

17 

X 

GL  23  Analyze  Schedule  and  Cost  Variances 

GL  26  Implement  Managerial  Actions 

18 

X 

GL  23  Analyze  Schedule  and  Cost  Variances 

GL  27  Develop  Revised  Estimates  of  Cost  at  Completion 
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We  based  our  decision  on  the  importance  of  accurately  identifying  EVM 
data  such  as  variances  in  cost  and  schedule.  The  EVM  data  obtained  in  this  criterion  are 
critical  to  measuring  the  health  status  of  a  program  (GAO,  2009).  Furthermore,  by 
requiring  a  thorough  analysis  of  the  causes  for  exceeding  a  given  threshold,  management 
can  better  develop  a  course  of  action  to  fix  the  variances.  GL  23  provides  the  data  to 
measure  program  status,  for  without  it,  there  is  no  EVM  to  measure,  thus  defeating  the 
purpose  and  benefits  of  the  EVMS. 

h.  EVM  Guideline  26:  Implement  Managerial  Actions 

In  this  comparison,  Table  15  shows  that  GL  26  was  selected  as  more 
severe  than  GL  27. 


Table  15.  GL  26  vs.  GL  27 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

19 

X 

GL  26  Implement  Managerial  Actions 

GL  27  Develop  Revised  Estimates  of  Cost  at  Completion 

Although  common  indicators  of  poor  performance  show  frequent 
deficiencies  in  revising  and  updating  EACs  in  accordance  with  the  PMB  as  described  in 
GL  27,  implementing  managerial  actions  as  described  in  GL  26  affects  how  EACs  are 
updated.  In  the  F-22  case,  GL  26  was  rated  as  insufficient  for  failure  to  take  necessary 
actions  to  realign  the  program  based  on  the  variances,  which  led  to  inaccurate  EACs 
(Dibert  &  Velez,  2006).  EACs  must  be  compared  to  the  PMB  on  a  continual  basis,  but  if 
the  necessary  managerial  actions  are  not  taken  to  reconfigure  the  PMB  based  on  any 
schedule  and  cost  variances,  EACs  are  to  be  calculated  accurately.  GAO  (2009)  showed 
that  many  programs  tend  to  develop  overly  optimistic  EACs,  have  no  reasonable  plan  to 
achieve  the  EAC,  and  fail  to  account  for  risks.  The  eCAR  database  shows  that  GL  27  is 
the  most  frequent  deficiency  at  13.7%  for  all  13  EVM  guidelines.  Given  that  GL  27  is 
highest  rate  of  deficiency  clearly  shows  the  importance  of  ensuring  GL  23  and  26  are 
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implemented  correctly  to  prevent  deficiencies  in  GL  27.  Failure  to  implement  corrective 
actions  result  in  poorly  defined  EACs;  therefore,  GL  26  poses  a  higher  risk  in  severity. 

i.  EVM  Guideline  28:  Incorporate  Changes  into  Plans,  Budgets, 
and  Schedules 

In  our  last  comparison,  we  chose  GL  28  to  be  more  severe  in  consequence 
than  GL  32.  The  intent  of  GL  28  is  to  realistically  update  the  PMB  so  that  frequent 
changes  to  program  are  prevented.  Incorporating  the  changes  in  a  timely  manner  ensures 
that  execution  of  changes  is  implemented.  On  the  other  hand,  documenting  the  changes 
sequentially  through  tracking  a  record  as  described  in  GL  32  should  occur  once  the 
changes  are  incorporated.  We  found  GL  32  to  be  more  a  routine  administrative  function 
that  bears  less  risk  if  not  executed. 


Table  16.  GL28vs.  GL  32 


Pairwise 

Comparison 

Selected 

EVM  Guidelines 

20 

X 

GL  28  Incorporate  Changes  into  Plans,  Budgets,  and 

Schedules 

GL  32  Document  Changes  to  the  PMB 

B.  RANK  ORDER  RESULTS 

Upon  ranking  the  four  EVMS  groups  and  conducting  the  pairwise  comparison  of 
the  EVMS  guidelines  within  their  respective  groups,  we  used  the  pairwise  scores  to 
determine  the  overall  ranking  for  all  13  EVMS  high  risk  guidelines  ranking  each 
guideline  from  1  as  most  severe  to  13  as  least  severe.  Table  7  illustrates  the  results  of  the 
guideline  pairwise  score  and  rank  order. 
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Table  17. 


EVMS  Guideline  Pairwise  Score  and  Rank  Order 


EVM 

Rank 

Order 

Group  Ranking 

Guidelines 

Pairwise 

Score 

#1 

#  1.  Planning, 
Scheduling,  & 
Budgeting 

8 

5 

#2 

6 

4 

#3 

7 

3 

#4 

10 

2 

#5 

9 

1 

#6 

12 

0 

#7 

#  2,  Analysis 

23 

2 

#8 

26 

1 

#9 

27 

0 

#  10 

#  3.  Organization 

1 

1 

#  11 

3 

0 

#  12 

#  4.  Revisions 

28 

1 

#  13 

32 

0 

Based  on  our  analysis  in  the  previous  section  and  implementation  of  our  rank 
order  methodology,  Table  18  lists  the  overall  rank  order  of  severity  for  the  13  EVMS 
high  risk  guidelines. 


Table  18.  Overall  Rank  Order  of  Severity 


Rank 

Order 

EVMS 
High  Ris  k 
Guidelines 

#  1 

GL  8 

#  2 

GL  6 

#  3 

GL  7 

#  4 

GL  10 

#  5 

GL  9 

#  6 

GL  12 

#  7 

GL  23 

#  8 

GL  26 

#  9 

GL  27 

#  10 

GL  1 

#  11 

GL  3 

#  12 

GL  28 

#  13 

GL  32 
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The  rank  order  of  severity  provides  a  means  for  the  ACO  and  his  or  her  staff  to 
focus  their  limited  resources  on  the  surveillance  of  high  risk  guidelines.  For  example, 
Table  18  suggests  that  GL  8  is  a  high  risk  priority  and  surveillance  should  be  conducted 
weekly  verses  quarterly  for  a  less  severe  guideline  such  as  GL  32. 

C.  APPLICATION  OF  THE  OPERATIONAL  VAR  MODEL  TO  EVMS 

As  discussed  in  Chapter  III,  the  operational  VaR  is  the  recommended  quantitative 
model  to  calculate  risk  for  a  given  contract  with  significant  EVMS  deficiencies.  This 
section  discusses  the  application  of  the  operational  VaR  model  for  two  different  deficient 
EVMS  scenarios.  For  both  examples,  we  chose  to  calculate  the  VaR  using  the  95%,  99%, 
and  99.9%  confidence  levels  based  on  historical  VaR  applications  demonstrating  that 
financial  risk  analysis  generally  ranges  from  95%  to  99.9%  (Navarrete,  2006).  However, 
the  DCMA  personnel  have  the  discretion  to  set  the  confidence  level  intuitively  based  on 
the  government’s  best  interest  and  specific  contract  situation. 

In  calculating  the  loss  frequency  of  each  deficient  guideline,  we  used  the  eCAR 
database  to  obtain  the  number  of  deficient  GLs  by  CAGE  code  per  year.  Given  the  data, 
we  developed  a  modified  method  to  obtain  the  loss  frequency  in  the  future.  Our  modified 
method  to  obtain  the  loss  frequency  consisted  of  multiplying  the  time  factor  by  the  CAR 
weight: 

E (n)  =  Time  Factor  x  CAR  Weight  (6) 

where 

Time  Factor  =  (CARs  per  year  x  Time  Remaining  on  Contract)  (7) 

The  number  of  CARs  per  year  was  calculated  by  dividing  the  total  deficiencies  by 
the  number  of  year(s)  CARs  were  received.  The  CAR  weight  was  determined  using  the 
eCAR  database  of  the  frequency  of  deficiencies  by  CAGE  code  per  year  and  obtaining  a 
probability. 

In  determining  the  loss  severity,  we  chose  to  use  both  the  DCMA  and  contractor’s 
administrative  and  labor  costs  involved  in  investigating  and  determining  the  root  cause  of 
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a  deficiency  and  processing  a  CAR.  The  costs  of  pursuing  a  CAR  involve  preparing  the 
corrective  action  plan;  implementing  the  corrective  action  plan;  independent  validation 
that  corrective  actions  were  actually  taken;  revalidation  of  the  EVMS  (if  necessary);  and 
closing  out  the  CAR.  Since  EVM  is  applied  to  all  incentive  and  cost  type  contracts  valued 
at  $20  million  or  above,  the  costs  to  run  the  EVMS  and  associated  costs  to  correct 
significant  EVMS  guideline  deficiencies  can  be  passed  by  the  contractor  to  the 
government  as  allowable  costs. 

In  addition,  any  cost  overruns  or  cost  variance  (CV)  that  a  specific  deficiency 
caused  to  the  contractor  and  government  would  also  be  taken  into  the  loss  severity 
calculation.  The  administrative  costs  of  fixing  a  CAR  and  cost  overruns  if  applicable 
would  show  that  more  costs  were  spent  for  work  accomplished  than  was  planned,  which 
represents  risk  to  the  government.  Because  we  did  not  have  data  on  CAR  severity  costs, 
we  used  fictitious  severity  data  for  both  EVMS  examples. 

The  operational  VaR  to  the  government  consist  of  both  EL  and  UL  at  a  given 
confidence  interval,  which  covers  the  government’s  expected  losses  and  severe 
unexpected  losses  from  a  deficient  guideline(s).  Figure  5  depicts  Navarette’s  (2006) 
illustration  of  how  both  EL  and  UL  at  the  99.9%  confidence  level  make  up  the 
operational  VaR  and  the  dollar  amount  of  risk  the  government  suffers  as  a  result  of  the 
deficiencies. 
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Each  example  follows  these  nine  steps: 

1.  Calculate  the  CAGE  Code  CAR  weight  for  each  guideline.  Add  the  new 
guideline  deficiency  to  the  current  amount  of  guideline  deficiencies,  then  divide  by  the 
updated  total  amount  of  deficiencies  from  each  guideline. 

CAR  Weight i  = 

( Current  Guideline  Frequency  +  New  Guideline  Frequency ) 

Total  Amount  of  Frequencies  from  each  Guideline  (8) 

2.  Calculate  the  time  factor.  Determine  the  CARs  per  year  (total  amount  of  CARs 
divided  by  total  amount  of  years  CARs  received)  multiplied  by  the  time  remaining  on 
contract  (equation  7). 

Time  Factor  =  (CARs  per  year  x  Time  Remaining  on  Contract)  (9) 

3.  Calculate  E (n),  the  loss  frequency  by  multiplying  the  time  factor  by  each  CAR 

weight. 
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E(«)i  =  Time  Factor  x  CAR  Weight  (10) 

4.  Obtain  E(x)i,  the  loss  of  severity.  Use  the  administrative  and  labor  costs 
required  to  process  each  deficient  guideline  and  cost  overruns,  if  applicable,  caused  by 
the  guideline  deficiency. 

5.  Calculate  EL]  for  each  guideline  deficiency,  Multiply  equation  (10)  by  E(x)j 
to  obtain  the  EL  value  for  each  guideline. 

EL]  =  E(n)i  x  E(x),  (1 1) 

6.  Calculate  the  overall  EL.  Multiply  the  mean  of  E (n)  and  E(x). 

Overall  EL  =  Mean  of  E(x)  x  Mean  of  E(n)  (12) 

7.  Calculate  the  operational  VaR.  Take  the  percentile  (established  confidence 
level)  of  the  loss  distribution  (range  of  ELs).  For  our  research,  we  calculated  the  95th, 
99th,  and  99.9th  percentile  using  the  Microsoft  Excel  PERCENTILE. INC  function. 

Operational  VaR  =  Percentile  Range  of  EL(s)  (13) 

8.  Calculate  the  overall  UL.  Subtract  the  Overall  EL  from  the  operational  VaR. 

Overall  UL=  Operational  VaR  -  Overall  EL  (14) 

9.  The  withhold  amount  caused  by  the  deficiencies  is  the  calculated  operational 
VaR  not  to  exceed  the  maximum  5%  of  contractor  progress  payments. 

1.  Scenario  1:  CAGE  Code  XX 

In  Scenario  1,  CAGE  code  XX  or  Contract  XX  has  been  issued  a  CAR  Level  III 
with  the  following  repeated  EVMS  guideline  deficiencies:  3,  6,  10,  12,  23,  and  27.  Using 
unidentified  EVM  data  from  the  Defense  Cost  and  Resource  Center  (DCARC)  database, 
Table  19  outlines  pertinent  EVM  data  for  Scenario  1  with  guideline  severity  costs  and 
any  cost  overruns  as  a  result  of  the  CAR. 
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Table  19.  Scenario  1  Information 


Scenario  1:  CAGE  Code  XX 

Duration 

(Mo.) 

62 

Time  Remaining 
on  Contract 
(Mo.) 

24 

Budget  at 
Completion 
(BAC) 

$  729,000,000.00 

Total  Years  of 
CARs  Received 

5 

* 

Calculated 
Progress 
Payments  $ 
per  month 

$  11,758,064.52 

Max  Withhold 
Amount  $  per 
month 

$  587,903.23 

GL  Severity  Dollars 

EVMS 

Deficient 

Guidelines 

Administrative  $ 
Costs 

Overrun  $  Costs 

Total  $  Loss 
Severity  E(x) 

3 

$  75,000.00 

$  400,000.00 

$  475,000.00 

6 

$  110,000.00 

$  300,000.00 

$  410,000.00 

10 

$  50,000.00 

$  200,000.00 

$  250,000.00 

12 

$  20,000.00 

$  0 

$  20,000.00 

23 

$  40,000.00 

$  600,000.00 

$  640,000.00 

27 

$  95,000.00 

$  1,200,000.00 

$  1,295,000.00 

*Calculated  using  BAC  divided  by  duration 


Executing  the  nine  steps,  the  operational  VaR  for  the  six  EVMS  deficiencies  is 
calculated  as  such: 

1.  Calculate  the  CAGE  Code  CAR  weight  for  each  guideline.  Add  the  new 
guideline  deficiency  to  the  current  amount  of  guideline  deficiencies,  then  divide  by  the 
updated  total  amount  of  deficiencies  from  each  guideline. 

Table  20  illustrates  that  guidelines  3,  6,  10,  12,  23,  and  27  are  repeated 
deficiencies,  which  caused  their  respective  CAR  weight  percentages  to  change. 
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Table  20.  Scenario  1:  Step  1  Calculations 


EVMS 

Guideline 

“Current” 
Cage  Code 
XX 

Frequency 

CAR 

Weight 

“New” 
Cage 
Code  XX 
Frequency 

New  CAR 
Weight 

1 

0 

0.000 

0 

0.000 

3 

1 

0.091 

2 

0.118 

6 

1 

0.091 

2 

0.118 

7 

0 

0.000 

0 

0.000 

8 

1 

0.091 

1 

0.059 

9 

0 

0.000 

0 

0.000 

10 

2 

0.182 

3 

0.176 

12 

1 

0.091 

2 

0.118 

23 

1 

0.091 

2 

0.118 

26 

1 

0.091 

1 

0.059 

27 

1 

0.091 

2 

0.118 

28 

2 

0.182 

2 

0.118 

32 

0 

0.000 

0 

0.000 

Total 

Deficiencies 

11 

New  Total 
Deficiencies 

17 

2.  Calculate  the  time  factor.  Determine  the  CARs  per  year  (total  amount  of  CARs 
divided  by  total  amount  of  years  CARs  received)  multiplied  by  the  time  remaining  on 
contract  (equation  7). 


50 


Table  21.  Scenario  1:  Step  2  Calculations 


CARs  per  year  =  (Total  Amount  of  CARS)  / 
(Total  Years  of  CARs  Received) 

Total 
Amount 
of  CARS 
Received 

Total 
Years  of 
CARs 
Received 

CARs  per 
year 

Time 

Remaining 

on 

Contract 

(yrs) 

17 

5 

3.4 

2 

Time  Factor  (CARs  per  year  X 
Time  Remaining  on  Contract) 

6.8 

3.  Calculate  E(n),  the  loss  frequency  by  multiplying  the  time  factor  by  each  CAR 
weight.  CAR  weights  were  taken  from  the  calculations  in  Table  20. 


Table  22.  Scenario  1:  Step  3  Calculations 


E(n)  =  Time  Factor 
x  CAR  Weight 

EVMS 

Deficient 

Guidelines 

Time 

Factor 

CAR 

Weight 

Loss 

Frequency 

E(n) 

3 

6.8 

0.118 

0.80 

6 

6.8 

0.118 

0.80 

10 

6.8 

0.176 

1.20 

12 

6.8 

0.118 

0.80 

23 

6.8 

0.118 

0.80 

27 

6.8 

0.118 

0.80 
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4.  Obtain  E(x)i,  the  loss  of  severity.  Use  the  administrative  and  labor  costs 
required  to  process  each  deficient  guideline  and  cost  overruns,  if  applicable,  caused  by 
the  guideline  deficiency.  See  Table  19. 

5.  Calculate  ELi  for  each  guideline  deficiency  by  multiplying  equation  (10)  by 
E(x)i  to  obtain  the  EL  value  for  each  guideline.  Loss  severity  and  loss  frequency  data 
obtained  from  Tables  19  and  22  respectively. 


Table  23.  Scenario  1:  Steps  4-6  Calculations 


EL  =  E( x)  x  E(n) 

EVMS 

Deficient 

Guidelines 

Loss  Severity  E(x) 

Loss 

Frequency 

E(n) 

Expected  Loss 
(EL) 

3 

$  475,000.00 

0.80 

$  381,140.00 

6 

$  410,000.00 

0.80 

$  328,984.00 

10 

$  250,000.00 

1.20 

$  299,200.00 

12 

$  20,000.00 

0.80 

$  16,048.00 

23 

$  640,000.00 

0.80 

$  513,536.00 

27 

$  1,295,000.00 

0.80 

$  1,039,108.00 

Mean  of  E(x) 

Mean  of  E(n) 

$  515,000.00 

0.87 

Mean  of  E(x)  x  Mean  of  E(n) 

Overall  EL  $448,050.00 

6.  Calculate  the  overall  EL.  Multiply  the  mean  of  E(n)  and  E(x).  See  Table  23. 

7.  Calculate  the  operational  VaR,  Take  the  percentile  (established  confidence 
level)  of  the  loss  distribution  (range  of  ELs).  For  our  research,  we  calculated  the  95th, 
99th,  and  99.9th  percentile  using  the  Microsoft  Excel  PERCENTILE. INC  function.  See 
Figure  6  and  Table  24. 
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Table  24.  Scenario  1:  Step  7  Calculations 


EVMS 

Deficient 

Guidelines 

Loss  Severity 
E(x) 

Loss  Frequency 
E(n) 

Expected  Loss 
(EL) 

3 

$ 

475,000.00 

0.80 

$  381,140.00 

6 

$ 

410,000.00 

0.80 

$  328,984.00 

Take  the  95th,  99th,  and  99.9th 

10 

$ 

250,000.00 

1.20 

$  299,200.00 

^r==— 

Percentile  of  the  Loss 

12 

$ 

20,000.00 

0.80 

$  16,048.00 

Distribution  (range  ofELs) 
using  the 

23 

$ 

640,000.00 

0.80 

$  513,536.00 

27 

$ 

1,295,000.00 

0.80 

$1,039,108.00 

_ - 

Function 

Mean  ofE(x) 

MeanofE(n) 

$ 

515,000.00 

0.87 

MeanofE(x)xMeanof  E(n) 

Overall  EL 

$448,050.00 

Table  25.  Scenario  1:  Steps  7-8  Calculations 


Percentiles 

Losses 

Confidence 

Level 

Operational 

VaR  (Percentile 
of  Loss 
Distribution) 

Overall  EL 
(Aggregate 

Loss 

Distribution) 

Overall  UL 
(Operational 
VaR  -  EL) 

95.00% 

$  907,715.00 

$  448,050.00 

$  459,665.00 

99.00% 

$  1,012,829.40 

$  448,050.00 

$  564,779.40 

99.90% 

$  1,036,480.14 

$  448,050.00 

$  588,430.14 

8.  Calculate  the  overall  UL.  Subtract  the  overall  EL  from  the  operational  VaR. 
See  Table  24  for  calculations.  Figure  7  is  a  depiction  of  the  loss  distribution  at  the  95th 
percentile  that  shows  how  the  overall  EL  and  UL  equate  to  the  operational  VaR. 
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9.  The  withhold  amount  caused  by  the  deficiencies  is  the  calculated  operational 
VaR  not  to  exceed  the  maximum  5%  of  contractor  progress  payments. 


Table  26.  Scenario  1 :  Total  VaR  and  Withhold  Amount  Determination 


VaR  at  95% 
Confidence 
Level 

$  907,715.00 

Monthly 

Progress 

Payments 

$11,758,064.52 

Monthly 

Maximum 

Withhold 

Amount 

$  587,903.23 

Final 

Amount  to 
Withhold 

$  587,903.23 
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Scenario  1  shows  that  the  total  operational  VaR  for  EVMS  guideline  deficiencies 
(3,  6,  10,  12,  23,  and  27)  is  approximately  $908,000.  The  monetary  risk  caused  by  the 
EVMS  deficiencies  exceeds  the  maximum  monthly  withhold  amount  of  approximately 
$600,000.  The  severity  of  the  six  EVMS  deficiencies  for  Contract  XX  were  equal  to  a 
progress  payment  withhold  of  7.7%,  which  exceeds  the  statutory  limit  of  withholding  a 
maximum  5%  of  progress  payments. 

2.  Scenario  2:  CAGE  Code  YY 

In  Scenario  2,  CAGE  code  YY  or  Contract  YY  has  been  issued  a  CAR  Level  III 
with  never  before  seen  EVMS  guideline  deficiencies  land  32.  Using  unidentified  EVM 
data  from  the  Defense  Cost  and  Resource  Center  (DCARC)  database,  Table  26  outlines 
pertinent  EVM  data  for  Scenario  2  with  guideline  severity  costs  and  any  cost  overruns  as 
a  result  of  the  CAR. 


Table  27.  Scenario  2  Information 


Scenario  2:  CAGE  Code  YY 

Duration 

(Mo.) 

41 

Time  Remaining 
on  Contract  (Mo.) 

24 

Budget  at 
Completion 
(BAC) 

$  198,000,000.00 

Total  Years  of 
CARs  Received 

2 

* 

Calculated 
Progress 
Payments  $ 
per  month 

$  4,829,268.29 

Max  Withhold 
Amount  $  per 
month 

$  241,463.41 

GL  Severity  Dollars 

EVMS 

Deficient 

Guidelines 

Administrative  $ 
Costs 

Overrun  $  Costs 

Total  $  Loss 
Severity  E(x) 

1 

$  60,000.00 

$  160,000.00 

$  220,000.00 

32 

$  15,000.00 

$ 

$  15,000.00 

*  Calculated  using  BAC  divided  by  duration 
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Executing  the  nine  steps,  the  operational  VaR  for  the  two  EVMS  deficiencies  is 
calculated  as  such: 

1.  Calculate  the  CAGE  Code  CAR  weight  for  each  guideline.  Add  the  new 
guideline  deficiency  to  the  current  amount  of  guideline  deficiencies,  then  divide  by  the 
updated  total  amount  of  deficiencies  from  each  guideline. 

Table  27  illustrates  that  guidelines  1  and  32  are  never  before  seen  deficiencies, 
which  caused  their  respective  CAR  weight  percentages  to  change. 


Table  28.  Scenario  2:  Step  1  Calculations 


EVMS 

Guideline 

“Current” 
Cage  Code 
XX 

Frequency 

CAR 

Weight 

“New” 
Cage 
Code  XX 
Frequency 

New  CAR 
Weight 

1 

0 

0.000 

1 

0.167 

3 

1 

0.250 

1 

0.167 

6 

0 

0.000 

0 

0.000 

7 

0 

0.000 

0 

0.000 

8 

0 

0.000 

0 

0.000 

9 

1 

0.250 

1 

0.167 

10 

0 

0.000 

0 

0.000 

12 

0 

0.000 

0 

0.000 

23 

0 

0.000 

0 

0.000 

26 

0 

0.000 

0 

0.000 

27 

1 

0.250 

1 

0.167 

28 

1 

0.250 

1 

0.167 

32 

0 

0.000 

1 

0.167 

Current 

Total 

Deficiencies 

4 

New  Total 
Deficiencies 

6 

2.  Calculate  the  time  factor.  Determine  the  CARs  per  year  (total  amount  of  CARs 
divided  by  total  amount  of  years  CARs  received)  multiplied  by  the  time  remaining  on 
contract  (equation  7). 
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Table  29.  Scenario  2:  Step  2  Calculations 


CARs  per  year  =  (Total  Amount  of  CARS)  / 
(Total  Years  of  CARs  Received) 

Total 

Amount 
of  CARS 
Received 

Total 
Years  of 
CARs 
Received 

CARs  per 
year 

Time 

Remaining 

on 

Contract 

(yrs) 

6 

2 

3 

2 

Time  Factor  (CARs  per  year  X 
Time  Remaining  on  Contract) 

6 

3.  Calculate  E(n),  the  loss  frequency.  Multiply  the  time  factor  by  each  CAR 
weight.  CAR  weights  were  taken  from  the  calculations  in  Table  27. 


Table  30.  Scenario  2:  Step  3  Calculations 


E(n)  =  Time  Factor 
x  CAR  Weight 

EVMS 

Deficient 

Guidelines 

Time 

Factor 

CAR 

Weight 

Loss 

Frequency 

E(n) 

1 

6 

0.167 

1.002 

32 

6 

0.167 

1.002 

4.  Obtain  E(x)i,  the  loss  of  severity,  use  the  administrative  and  labor  costs 
required  to  process  each  deficient  guideline  and  cost  overruns,  if  applicable,  caused  by 
the  guideline  deficiency.  See  Table  26. 

5.  Calculate  ELi  for  each  guideline  deficiency.  Multiply  equation  (10)  by  E(x)i 
to  obtain  the  EL  value  for  each  guideline.  Loss  severity  and  loss  frequency  data  obtained 
from  Tables  26  and  29  respectively. 
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Table  3 1 .  Scenario  2:  Steps  4-6  Calculations 


EL  =  E(x)  x  E(n) 

EVMS 

Deficient 

Guidelines 

Loss  Severity  E(x) 

Loss 

Frequency 

E(n) 

Expected  Loss 
(EL) 

1 

$  220,000.00 

1.002 

$  220,440.00 

32 

$  15,000.00 

1.002 

$  15,030.00 

Mean  of  E(x) 

Mean  of  E(n) 

$  117,500.00 

1.002 

Mean  of  E(x)  x  Mean  of  E(n) 

Overall  EL 

$117,735.00 

6.  Calculate  the  overall  EL.  Multiply  the  mean  of  E(n)  and  E(x).  See  Table  30. 

7.  Calculate  the  operational  VaR.  Take  the  percentile  (established  confidence 
level)  of  the  range  of  EL(s).  For  our  research,  we  calculated  the  95th,  99th,  and  99.9th 
percentile  using  the  Microsoft  Excel  PERCENTILE. INC  function.  See  Figure  7  and 
Table  31. 


EVMS 

Deficient 

Guidelines 

Loss  Severity 
E(x) 

Loss  Lrequency 
E(n) 

Expected  Loss 
(EL) 

~T 

Take  the  95th,  99th,  and  99.9th 
Percentile  ofthe  Loss 
Distribution  (range  ofELs) 
using  the 

Microsoft  Excel  Percentile 
Function 

1 

S  220,000.00 

1.002 

S  220,440.00 

32 

S  15,000.00 

1.002 

S  15,030.00 

J" 

Figure  7.  Scenario  2:  Step  7  Calculations 
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Table  32.  Scenario  2:  Steps  7-8  Calculations 


Percentiles 

Losses 

Confidence 

Level 

Operational 

VaR  (Percentile 
of  Loss 
Distribution) 

Overall  EL 
(Aggregate 

Loss 

Distribution) 

Overall  UL 
(Operational 
VaR  -  EL) 

95.00% 

$  210,169.50 

$  117,735.00 

$  92,434.50 

99.00% 

$  218,385.90 

$  117,735.00 

$  100,650.90 

99.90% 

$  220,234.59 

$  117,735.00 

$  102,499.59 

8.  Calculate  the  overall  UL.  Subtract  the  overall  EL  from  the  operational  VaR. 
See  Table  31  for  calculations.  Figure  9  is  a  depiction  of  the  loss  distribution  at  the  95th 
percentile  that  shows  how  the  overall  EL  and  UL  equate  to  the  operational  VaR. 


Loss  Distribution  (5)  95th  Percentile 

- - V 

EL  =  $117,735 

- > 

VaR  =  $210,170 

‘"c' 

5*  0.15 

c 

a> 

3 

O’ 

fl) 

GL  32 

l/l 

«/» 

O 

—i 

0.00 

$3,006.00  $183,766.80 

Loss  Severity  E(x) 

Figure  8.  Scenario  2:  Loss  Distribution  Chart  at  95th  Percentile 


9.  The  withhold  amount  caused  by  the  deficiencies  is  the  calculated  operational 
VaR  not  to  exceed  the  maximum  5%  of  contractor  progress  payments. 
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Table  33. 


Scenario  2:  Total  VaR  and  Withhold  Amount  Determination 


VaR  at  95% 
Confidence 
Level 

$  210,169.50 

Monthly 

Progress 

Payments 

$  4,829,268.29 

Monthly 

Maximum 

Withhold 

Amount 

$  241,463.41 

Final 

Amount  to 
Withhold 

$  210,169.50 

Scenario  2  shows  that  the  total  operational  VaR  for  EVMS  guideline  deficiencies 
(1  and  32)  is  approximately  $210,170.  In  this  case,  the  risk  caused  by  the  EVMS 
deficiencies  did  not  exceed  the  maximum  withhold  amount  of  $241,463.  The  severity  of 
the  two  EVMS  deficiencies  for  Contract  YY  equaled  to  a  progress  payment  withhold  of 
approximately  4.4%. 

D.  SUMMARY 

In  this  chapter,  we  have  analyzed  the  severity  for  each  of  the  13  high  risk  EVMS 
guidelines  and  developed  a  rank  order  of  severity.  By  obtaining  a  rank  order  of  severity, 
we  were  able  to  assign  each  guideline  with  a  weighted  risk  factor  which  ACOs  can  use  in 
prioritizing  their  limited  resources  on  their  surveillance  of  high  risk  guidelines.  In 
scenario  1,  the  operational  VaR  exceeded  the  federal  maximum  withhold  amount.  In 
scenario  2,  the  operational  VaR  was  below  the  maximum  withhold  amount.  Both 
scenarios  demonstrate  that  ACOs  can  benefit  by  objectively  justifying  withhold  amounts. 

The  next  chapter,  Conclusion,  answers  each  research  objective,  provides 
recommendations  for  the  DCMA,  and  discusses  our  research  limitations. 
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VI.  SUMMARY,  CONCLUSION  AND  FURTHER  RESEARCH 


A.  SUMMARY 

In  Chapter  I,  we  discussed  the  purpose  of  our  research  by  explaining  the 
Pentagon’s  new  business  systems  rule  and  our  intent  to  focus  solely  on  EVMS.  The  new 
business  system  rule  addresses  deficient  contractor  business  systems  and  the  DCMA’s 
authority  to  withhold  5%  of  contractor  progress  payments  when  a  significant  deficiency 
is  present.  Contractors  are  held  more  accountable  and  punished  monetarily  for  failures  to 
comply  with  EVMS  guidelines  under  the  new  business  rule.  Our  research  efforts  benefit 
the  DCMA  by  recommending  a  quantitative  risk  analysis  tool  that  can  be  used  by  ACOs 
to  justify  the  amount  of  payment  withholds. 

In  Chapter  II,  we  provided  an  in-depth  overview  of  the  13  high  risk  EVMS 
guidelines  that  DCMA  EVM  specialists  asked  us  to  focus  on.  This  chapter  laid  the 
foundation  to  understand  root  causes  and  severity  of  EVMS  significant  deficiencies.  We 
used  case  examples  from  the  F-22  Spiral  2  program,  GAO  report  on  cost  estimation  and 
assessment,  and  EVMS  subject  matter  experts  in  providing  common  examples  of 
deficiencies  for  each  of  the  guidelines. 

In  Chapter  III,  we  described  quantitative  risk  analysis  in  DoD  acquisitions.  This 
chapter  provided  key  concepts  in  answering  our  research  objectives.  ACOs  can  calculate 
the  risk  to  the  government  by  applying  the  operational  VaR  model  to  justify  the  amount 
to  withhold  from  contractor  progress  payments.  The  operational  VaR  model  is 
recommended  because  it  takes  into  account  the  frequency  of  the  loss  event  occurring  and 
the  severity  of  loss  in  calculating  risk.  We  also  explored  the  pairwise  comparison 
methodology  to  assist  us  in  developing  a  rank  order  of  the  13  high  risk  EVMS  guidelines. 

In  Chapter  IV,  we  discussed  the  eCAR  database  and  obtained  the  loss  frequency 
of  the  high  risk  guidelines,  which  is  required  for  the  application  of  the  operational  VaR 
model.  Establishing  the  rank  order  of  severity  for  the  EVM  guidelines  was  conducted  in 
three  stages:  Rank  ordering  of  the  EVM  group,  using  the  pairwise  comparison  score  to 
rank  order  the  corresponding  guidelines  within  each  group,  and  then  combining  both 
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EVM  group  and  guideline  rankings  to  obtain  the  overall  ranking  from  1  to  13.  The  risk 
factors  for  each  rank  item  were  established  using  the  rank  sum  methodology,  which 
provide  ACOs  a  means  to  prioritize  their  surveillance  activities. 

In  the  last  chapter,  we  analyzed  the  risk  severity  that  each  guideline  posed  and 
developed  a  rank  order  of  severity  for  each  of  the  13  guidelines.  Upon  obtaining  a  rank 
order  for  each  guideline,  we  applied  the  operational  VaR  method  to  two  different  EVMS 
guideline  significant  deficiency  scenarios. 

B.  CONCLUSION 

The  intent  of  this  project  was  to  objectively  and  quantitatively  portray  EVMS  risk 
in  a  way  that  supports  a  monetary  withhold  decision  and  can  withstand  objection  (to 
include  litigation)  from  the  defense  contractor.  In  this  project,  we  hypothesize  that  using 
quantitative  risk  models  such  as  the  operational  VaR  method  and  simplification  of  that 
business  model  for  use  by  contracting  officers  provides  value  to  DCMA  professionals  in 
the  implementation  of  the  new  business  rule.  Our  project  focused  on  providing  the 
DCMA  with  a  more  defensible  risk  value  model  as  the  basis  for  withholding  contractor 
payments.  In  order  to  assist  our  research  efforts,  there  were  three  project  objectives.  The 
following  are  the  findings  and  recommendations  associated  with  each  research  objective: 

1.  Determine  whether  the  13  EVMS  “high  risk”  guidelines  can  be 
grouped  with  respect  to  root  causes  (causality  of  risk). 

a.  Findings 

This  research  objective  required  us  to  evaluate  the  rank  or  natural  order  to 
the  potential  severity  of  the  deficiency  posed  by  these  guidelines  and  to  assess  the  degree 
of  inter-dependence  or  causality  across  the  13  critical  EVMS  guidelines.  We  found  that 
the  high  risk  guidelines  were  already  assembled  within  an  EVM  group  for  which  the 
group’s  function  served  as  associated  root  causes  of  risk  as  shown  in  Table  7. 

By  first  ranking  the  severity  of  the  EVM  group,  each  guideline  fell  into  a  category 
of  risk  for  a  program.  By  conducting  a  pairwise  comparison,  we  were  then  able  to  rank 
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order  each  guideline  within  its  respective  EVM  group  to  detennine  the  overall  rank  order 
of  severity  as  shown  in  Table  17. 

b.  Recommendations 

Using  our  rank  order  methodology,  the  DCMA  could  conduct  a  more 
formal  method  of  rank  ordering  high  risk  guidelines  based  on  AGO  and  EVM  subject 
matter  experts  within  this  field.  An  accurate  rank  order  of  GL  severity  is  essential  to 
prioritizing  the  DCMA’s  limited  resources  to  fixing  and  monitoring  the  most  severe  GLs. 

2.  Evaluate  which  quantitative  method(s)  can  be  used  to  calculate  risk 

value  with  respect  to  non-compliance  with  both  critical  and  non- 

critical  guidelines. 

a.  Findings 

In  this  research  objective,  we  found  the  operational  VaR  model  to  be 
applicable  in  calculating  risk  value  of  non-compliant  guidelines  and  identifying 
quantitative  definition  of  significant  deficiency.  In  Chapter  V,  there  are  two  scenarios  in 
which  we  calculate  the  monetary  risk  to  the  government  in  nine  steps.  By  obtaining  a  loss 
frequency  and  a  loss  severity  for  each  significant  deficiency,  we  were  able  to  calculate 
the  total  operational  VaR  for  EVMS  deficiencies.  We  found  that  the  monetary  amount  to 
withhold  differs  based  on  the  EVMS  significant  deficiencies  or  guidelines. 

b.  Recommendations 

The  nine  steps  recommended  for  calculating  the  operational  VaR  are  a 
valuable  quantitative  risk  analysis  tool  that  the  DCMA  could  implement  in  determining 
or  justifying  withhold  amounts  for  EVMS  deficiencies.  The  quantitative  model  is 
objective  and  removes  any  type  of  subjective  discretions  that  an  ACO  may  consider  in 
the  amount  to  withhold  from  the  contractor.  In  the  event  EVMS  deficiencies  calculate  to 
more  than  5%  of  progress  payments,  ACOs  will  have  the  confidence  to  withhold  the 
maximum  allowed.  Conversely,  if  EVMS  deficiencies  only  equate  to  3%  of  withholds, 
the  contractor  will  feel  confident  knowing  that  the  withhold  amount  was  not  based  on  the 
discretion  of  the  ACO  but  rather  an  established  quantitative  risk  analysis  tool. 
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3.  Determine  the  relationship  of  risk  value  calculations  and  findings  of 
EVMS  non-compliance  with:  (a)  probability  of  error,  (b)  magnitude 
of  errors,  and  (c)  adverse  impact  of  errors 

a.  Findings 

By  gathering  data  from  the  eCAR  database,  we  were  able  to  obtain  the 
probability  of  error,  also  known  as  loss  frequency,  used  in  calculating  the  expected  loss. 
By  using  the  contractor  and  government’s  administrative  costs  of  pursuing  and 
processing  a  CAR  to  include  cost  overruns  as  a  result  of  the  deficiencies,  we  were  able  to 
obtain  the  magnitude  of  error  (loss  severity).  The  mean  of  the  loss  severity  was 
multiplied  by  the  mean  of  the  loss  frequency  to  obtain  the  overall  expected  loss,  which 
allowed  us  to  obtain  the  operational  VaR  to  the  government,  also  known  as  the  adverse 
impact  of  errors. 

We  found  that  by  using  the  operational  VaR  formula,  we  are  able  to 
calculate  EVMS  non-compliant  risks  by  obtaining  the  probability,  magnitude,  and 
adverse  impact  of  error  to  obtain  a  monetary  risk  value.  This  research  objective  required 
us  to  develop  a  deterministic  rule  set  that  yields  a  consistent  and  repeatable  finding  of 
significant  deficiency.  Thus,  we  found  the  nine  steps  to  calculate  the  operational  VaR  as 
the  rule  set  that  objectively  and  consistently  yields  the  risk  value  of  EVMS  significant 
deficiencies. 


b.  Recommendations 

By  using  the  nine  steps  as  listed  in  calculating  the  operational  VaR  for 
EVMS  deficiencies,  ACOs  can  be  confident  in  withholding  calculated  monetary  amounts. 
Contractors  will  understand  the  importance  of  correcting  severe  deficiencies.  Severe 
deficiencies  hold  a  higher  monetary  risk  amount  to  the  government  and  vice  versa. 
Furthennore,  this  detenninistic  rule  set  can  also  be  used  as  a  guide  for  corrective  action 
enforcement  by  putting  a  calculated  withhold  value  on  each  CAR  level  I  or  II  to  warn 
contractors  of  the  potential  payment  withholds  that  might  come  with  a  CAR  level  III  or 
IV. 
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C.  RESEARCH  SHORTCOMINGS 

Every  effort  has  been  made  to  gather  information  for  an  accurate  and  thorough 
severity  analysis  of  each  guideline.  Our  initial  methodology  included  conducting  a 
pairwise  comparison  of  the  high  risk  guidelines  through  a  survey  distributed  to  Divisional 
Administrative  Contracting  Officers  (DACO),  ACOs,  and  EVM  subject  matter  experts. 
Due  to  the  DCMA’s  legal  constraints,  the  survey  was  not  approved  in  a  timely  manner. 
Professional  input  from  DACOs,  ACOs,  and  DCMA  EVM  subject  matter  experts  would 
have  assisted  in  rank  ordering  the  EVM  groups  and  guidelines. 

We  recommend  that  in  the  future  the  DCMA  conduct  a  fonnal  rank  order  analysis 
of  their  high  risk  guidelines  based  on  the  experience  and  working  knowledge  of  their 
personnel  in  order  to  develop  a  more  accurate  risk  factor  for  each  guideline.  The  survey 
we  developed  is  still  available  for  use  by  DCMA  as  shown  in  the  appendix. 

Secondly,  because  we  intended  to  distribute  a  survey,  we  decided  to  keep  the 
number  of  questions  limited  in  order  to  encourage  responses.  Thus,  we  only  conducted  a 
pairwise  comparison  among  guidelines  within  the  same  group  to  limit  the  questions  to  2 1 . 
However,  a  more  accurate  pairwise  comparison  should  have  compared  all  guidelines  to 
one  another,  which  would  have  resulted  in  78  comparisons  or  questions.  Furthermore,  our 
pairwise  methodology  limited  our  rank  order  analysis  to  ranking  guidelines  only  by  the 
rank  of  their  respective  EVM  group. 

Third,  we  recommend  that  the  eCAR  database  include  both  the  government  and 
contractor’s  administrative  and  labor  costs  associated  with  pursuing  and  processing  a 
CAR  to  completion  to  include  any  cost  overruns  that  the  deficiencies  cited  in  the  CAR 
created  for  the  government.  These  costs  are  considered  the  loss  severity  to  the 
government  and  essential  in  calculating  an  accurate  VaR  to  the  government. 

Last,  due  to  competition-sensitive  information,  we  were  unable  to  interview  the 
ACO  for  the  DCMA  at  Lockheed  Martin  (LM)  in  Sunnyvale,  CA.  This  DCMA  branch 
was  one  of  two  branches;  the  other  was  LM  in  Fort  Worth,  Texas,  that  has  withheld 
contractor  progress  payments  since  the  implementation  of  the  new  business  rule.  Insight 
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into  the  actual  circumstances  of  what  caused  the  ACO  to  withhold  payments  and  at  what 
percentage  could  have  highly  contributed  to  our  research  efforts. 

Despite  our  research  shortcomings,  the  information  gathered  and  analyzed  lays 
out  the  foundation  to  developing  a  more  accurate  rank  order  analysis  that  can  be  used  to 
realistically  calculate  the  operational  VaR  and  withhold  amount.  At  the  conclusion  of  this 
project,  we  still  met  our  research  objectives  and  validated  that  the  operational  VaR  model 
can  be  used  as  a  defensible  risk  value  model  as  the  basis  for  withholding  contractor 
payments. 

D.  AREAS  FOR  FURTHER  RESEARCH 

During  the  course  of  the  project,  we  identified  several  areas  for  further  research 
regarding  EVMS  deficiencies  and  the  VaR  model.  For  interested  researchers,  we 
recommend  the  following  for  future  areas  of  research: 

•  We  recommend  evaluating  the  root  causes  of  deficiencies  for  the 
remaining  19  guidelines  (2,  4,  5,  11,  13,  14,  15,  16,  17,  18,  19,  20,  21,  22, 
24,  25,  29,  20,  and  31)  that  were  not  included  in  this  research.  We  also 
recommend  ranking  all  32  guidelines  for  severity  by  experienced  EVM 
specialists. 

•  We  recommend  obtaining  contract  data  for  both  LM  Sunnyvale  and  LM 
Fort  Worth  and  apply  the  VaR  model  to  determine  if  there  were  any 
differences  in  a  quantitative  verses  a  subjective  withhold  amount 
determination.  Once  this  is  completed,  the  operational  VaR  model  should 
be  verified,  validated,  and  accredited  (VV&A)  by  an  independent  agency, 
The  Director,  Cost  Assessment  and  Program  Evaluation  (CAPE)  might  be 
the  correct  level  of  VV&A. 

•  Future  research  should  include  a  sensitivity  analysis  of  the  risk  factor 
determination,  pairwise  comparison,  and  VaR  model  to  recommend 
modifications  or  additions  to  the  model.  Sensitivity  analysis  would 
examine  the  sensitivity  of  inputs  to  the  VaR  model  based  on  changes  to 


the  guidelines  that  are  deficient. 
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•  Other  areas  for  research  include  searching  for  best  practices  from  private 
industries  in  evaluating  risk.  Can  risk  management  models  from  other 
private  industries  be  applied  to  the  business  system  deficiencies?  The 
variance-covariance  method,  Monte  Carlo  simulation,  risk  metrics,  and 
other  quantitative  approaches  can  be  researched  to  find  the  most  efficient 
VaR  model. 
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APPENDIX.  13  EVMS  HIGH  RISK  GUIDELINE  (GL)  SURVEY 


The  purpose  of  this  survey  is  to  evaluate  the  rank  or  natural  order  to  the  potential  severity 
of  the  deficiency  posed  by  the  selected  13  EVM  high  risk  guidelines 
(1,3,6,7,8,9,10,12,23,26,27,28,32).  This  research  project  excludes  the  accounting  EVM 
category  and  guidelines  and  focuses  on  only  13  guidelines  that  Senior  DCMA  Earned 
Value  Management  specialists  have  identified  as  high-risk  guidelines.  We  are  also 
assessing  the  degree  of  inter-dependence  or  causality  across  these  critical  EVMS 
guidelines.  The  data  we  collect  from  the  survey  will  assist  in  our  research  efforts  of 
objectively  and  quantitatively  portraying  EVMS  risk  in  a  way  that  supports  a  monetary 
withhold  decision  from  the  defense  contractor. 

You  are  invited  to  participate  in  a  research  study  titled  Deficient  Contractor  Business 
Systems:  Applying  the  Value  at  Risk  (VaR)  Model  to  Earned  Value  Management 
Systems.  We  would  like  you  to  participate  in  an  online  survey.  This  survey  will  consist  of 
21  questions.  During  the  survey  you  will  be  asked  to  provide  infonnation  about  your 
experiences,  subject  matter  expertise,  and  professional  opinion  in  regards  to  the  EVMS 
process.  Participation  in  this  survey  is  only  voluntary  and  any  questions  you  do  not 
answer  will  be  respected. 

There  are  2 1  questions  in  this  survey  and  will  take  no  longer  than  20  minutes  to 
complete. 
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Rank  Order  of  the  EVM  Categories 


Based  on  your  experience,  rank  order  the  selected  EVM  Categories  (Organization,  Planning, 
Scheduling,  and  Budgeting,  Analysis,  or  Revisions)  from  most  severe  to  least  severe  in 
deficiency. 

Each  category  should  be  evaluated  in  terms  of  the  severity,  which  is  the  consequence  of  the 
future  occurrence  of  a  significant  deficiency.  DFARS  Clause  252.234-7002  (a)  defines  a 
significant  deficiency  as  a:  shortcoming  in  the  system  that  materially  affects  the  ability  of 
officials  of  the  Department  of  Defense  to  rely  upon  information  produced  by  the  system  that  is 
needed  for  management  purposes. 

EVM  Categories 

Organization  (GL  1  and  3)  defines  the  range  of  requirements  prior  to  the  project  commencing. 
Planning,  Scheduling,  and  Budgeting  (GL  6,  7,  8,  9,  10,  and  12)  requires  a  management  control 
system  that  links  the  formal  planning,  scheduling,  and  budgeting  of  a  project  into  a  performance 
measurement  baseline  (PMB).  This  group  of  criterions  establishes  a  project  baseline  that  allows  for  a 
formal  means  of  project  discipline  and  assessment. 

Analysis  (GL  23,  26,  and  27)  requires  routine  submission  of  EVM  data  such  as  cost  and  schedule 
variances  to  maintain  effective  project  management.  Variances  should  be  evaluated  and  mitigated 
with  a  corrective  action  to  minimize  the  negative  impacts  on  the  project. 

Revisions  (GL  28  and  32)  require  approved  changes  to  the  project  in  a  timely  manner  to  allow  for 
integration. 

Click  on  an  item  in  the  list  on  the  left,  starting  with  your  highest  ranking  item,  moving  through  to  your  lowest 
ranking  item. 

Your  choices 


•  Group  1:  Organization 

•  Group  2:  Planning,  Scheduling,  and  Budgeting 

•  Group  4:  Analysis 

•  Group  5:  Revisions 
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Pairwise  Comparison  of  the  13  High  Risk  EVM  Guidelines 

For  each  of  the  20  questions  below,  select  the  guideline  (1,  3,  6,  7,  8,  9,  10,  12,  23,  26,  27,  28,  or 
32)  that  poses  a  higher  risk  defined  as  a  measure  of  future  uncertainties  in  achieving  program 
performance  goals  within  defined  cost  and  schedule  constraints. 

Each  guideline  (significant  deficiency)  should  be  evaluated  in  terms  of  the  severity,  which  is  the 
consequence  of  the  future  occurrence  of  that  significant  deficiency.  DFARS  Clause  252.234- 
7002  (a)  defines  a  significant  deficiency  as  a:  shortcoming  in  the  system  that  materially  affects 
the  ability  of  officials  of  the  Department  of  Defense  to  rely  upon  information  produced  by  the 
system  that  is  needed  for  management  purposes. 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 

Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  1  Define  Authorized  Work  (WBS  Elements) 
GL  3  Integrate  the  System 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 


Choose  one  of  the  following  answers 


GL  6  Schedule  the  Work 


GL  7  Identify  Products,  Milestones,  and  Indicators 
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Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  6  Schedule  the  Work 


•  GL  8  Plan  the  Performance  Measurement  Baseline 

(PMB) 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


u 


sn 


GL  6  Schedule  the  Work 

GL  9  Establish  Budgets  for  Work 


Please  enter  your  comment  here: 


“1 

nfl 


U 
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hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  6  Schedule  the  Work 


GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  6  Schedule  the  Work 

GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 
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Choose  one  of  the  following  answers 


GL  7  Identify  Products,  Milestones,  and  Indicators 


•  GL  8  Plan  the  Performance  Measurement  Baseline 

(PMB) 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  7  Identify  Products,  Milestones,  and  Indicators 


GL  9  Establish  Budgets  for  Work 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  7  Identify  Products,  Milestones,  and  Indicators 
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GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


u 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  7  Identify  Products,  Milestones,  and  Indicators 


GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here:  — 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


w  GL 
(PMB) 


Plan  the  Performance  Measurement  Baseline 


GL  9  Establish  Budgets  for  Work 
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Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


•  GL  8  Plan  the  Performance  Measurement  Baseline 
(PMB) 

•  ^  GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Please  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk. 

Choose  one  of  the  following  answers 


GL  8  Plan  the  Performance  Measurement  Baseline 
(PMB) 


GL  12  Identify  and  Control  Level  of  Effort 
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Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 


GL  9  Establish  Budgets  for  Work 


GL  10  Identify  Work  Packages 


Please  enter  your  comment  here: 


JJ 


LlS 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 


GL  9  Establish  Budgets  for  Work 

GL  12  Identify  and  Control  Level  of  Effort 
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Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 


GL  10  Identify  Work  Packages 


GL  12  Identify  and  Control  Level  of  Effort 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 


•  GL  23  Analyze  Schedule  and  Cost  Variances 

•  ^  GL  26  Implement  Managerial  Actions 


Please  enter  your  comment  here: 


1 


u 
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hoose  only  1  guideline  from  the  following  pair 


Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 


•  ®  GL  23  Analyze  Schedule  and  Cost  Variances 

•  ^  GL  27  Develop  Revised  Estimates  of  Cost  at 
Completion 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 

Choose  one  of  the  following  answers 


U 


•  ^  GL  26  Implement  Managerial  Actions 

•  ^  GL  27  Develop  Revised  Estimates  of  Cost  at 
Completion 


Please  enter  your  comment  here: 


hoose  only  1  guideline  from  the  following  pair 

Select  which  guideline  you  think  is  a  higher  risk  between  the  pair? 
Will  you  also  provide  your  professional  input  on  why  you  chose  the 
guideline  to  be  a  higher  risk? 
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Choose  one  of  the  following  answers 


•  28  Incorporate  Changes  into  Plans,  Budgets,  and 
Schedules 

•  ^  32  Document  Changes  to  the  PMB 


Please  enter  your  comment  here: 

hoose  only  1  guideline  from  the  following  pair 
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